Based on DevOps tool chain design process and before and after effect comparison

Click on "ZTE Developer Community" above and follow us

Read a first-line developer original good article every day

Status quo background

A project is a project launched to cooperate with large-scale video operation and maintenance. The requirements and task management remain on the original ts. The project relies on svn for code management and integrates with jenkins to realize continuous integration. The testing method is biased to supplement manual testing with special tools. In the early stage of the project, there are often difficulties in delivery of the version or the quality of the delivered version submitted for system testing cannot meet the requirements of the 1 round of release. Developers and testers are struggling to cope with demand adjustment and troubleshooting after the version submission. The code review is limited by physical space and time. It cannot be confirmed on a line-by-line basis. There is no overall framework for automated testing. Various special tools cannot be integrated or expanded. Tools such as code static inspection tools that help improve the quality of versions and the efficiency of development processes are not used.

solution

After researching and drawing lessons from DevOps' excellent experience of each company's brother projects and combining it with the actual situation of a project, after continuous evolution, TFS was finally formed as a demand and task management platform, and Gerrit code audit server was used as the core to build integrated code. Complete code flow for submission, concurrent testing, static inspection, review, feedback, and inventory entry. Introducing layered cloud CI to achieve code submission review ValidCI, code merged controlled library MergeCI, daily code quality inspection, DailyCI 3 quality protection , Version continuous iteration and grayscale grayscale release in the existing network, to achieve the current network operation and maintenance data collection and improve our probe acquisition modeling model based on the collected data.

1. End-to-end delivery panorama

2. Visualization of requirements tasks

TFS-assisted agile processes to track requirements, user stories, and Task completion

3. Cloud Code Hosting

In the project agile iterative evolution process, an important part is code management. The original use of svn code to manage the download speed is very slow, and it depends heavily on the server's hardware resource configuration. The code review under the code submission line is rather crude, and it needs to be manually merged after review. Errors and inefficiencies. After a comparative study, the use of GIT tools to manage source code not only improves code acquisition efficiency, but also cooperates with Gerrit to open code review functions and improve integration efficiency.

The main difference between Git and other version control systems is that Git only cares about whether the overall file data changes, while most other systems only care about the specific differences in the file content, but do not save these before and after changes in the difference data. In fact, Git is more like taking a snapshot of a changed file and recording it in a tiny file system. Each time an update is submitted, it takes an overview of the fingerprint information for all files and takes a snapshot of the file, and then saves an index pointing to the snapshot. To improve performance, if the file does not change, Git will not save it again. Instead, it will only make a link to the last saved snapshot.

Most operations in Git only need to access local files and resources without networking. Because Git keeps historical updates of all current projects on the local disk, it can be processed quickly. For example, if you want to browse the historical update summary of a project, Git doesn't need to go to an outside server to fetch the data back, but read it directly from the local database and show it to you. So you can read it right away without waiting. If you want to see the difference between the current version of the file and the one-month-old version, Git will take the one-month snapshot and the current file for a difference instead of asking the remote server to do it, or The old version of the file is pulled to the local for comparison.

4. Branch management strategy

The original branch strategy is to adopt the branch version for different market customization needs. The public requirements are used in the development process and the main line is used. When the specific market is released, the branching method is used, and the code synchronization takes too much time. In the extreme case, a common bug fix needs to synchronize to the same time. An active branch, and the CI launches just cover the mainline issue and it is easy to leak. After switching to git, the project is adjusted to take a single master backbone, and a small number of temporary Feature branches are supplemented by the development model. The backbone guarantees basic functions through the combination of automated testing and manual testing. The code branches are available in real time and can be released on demand. For the small-variation-based short-cycle Feature developed on the master, it is developed based on the master pull branch. After the self-test passes, the master merges:

5. Code review

Gerrit is a free, open source code review software that uses a web interface. Using the browser, the developers of the same team can review each other's modified program code and decide whether they can submit, return or continue to modify. The company's technical department provides Git and Gerrit support in a unified manner. In order to make full use of Git's features, the project's code is hosted on the company's cloud server, and through Gerrit, the code between developers is mutually audited. It solves the problem of inaccurate records, manual distortions, or distortions in manual walk-through surveys. The following figure shows the use of Gerrit renderings:

6. Code coverage detection

In the function code development process and test case development process, we introduced the gcov code coverage detection tool. Through its lcov generated scan report, as shown in the following figure:

The report clearly indicates the line-level coverage of the code, including: 1. The frequency of execution of each line of code; 2. Actually which code is actually executed; 3. Time spent (execution time) of each section code

According to the above report, you can clearly see whether the code is redundant or the test case design is not fully considered, resulting in low coverage, and then targeted analysis and optimization and improvement.

7. Continuous integration

In order to improve the output quality of the code, first of all, a certain part of the automated testing process was combed. The previously scattered special tools were reconstructed through the framework framework based on the RF (RobotFrameware) test, and a hierarchical cloud CI was introduced to perform the separation in different dimensions. Layer code quality inspection. Cloud CI implements continuous integration of VerifyCI, MergeCI, and Daily CI through the pipeline, making the code base change through the pipeline to the component automated scan detection and timely feedback through the mail push method, indicating the following diagram:

1) Continuous Integration - VerifyCI

When the developer debugs the self-test code and submits it to the gerrit managed library, it will automatically trigger VerfyCI, perform compile detection, code circle complexity scan, UT test, etc., and perform VerifyCI+2 auto-participate code review, and automatically push results through email. . After the developers develop the code, the static code, loop complexity check, and UT test can be performed immediately on the personal development environment in the personal development environment. The problem can be revealed before the gerrit review library is submitted. Get feedback on code quality. The actual effect is shown in the figure below:

2) Continuous Integration - MergeCI

After the review is passed, git is allowed to be stored. After Merge is submitted, code compilation detection, code circle complexity check, commercial version of KW static code scan, FT function test, etc. will be triggered, and the execution result of each pipeline plug-in will be emailed. The method is pushed to project-related members in real time, and the execution results are clear at a glance:

3) Continuous Integration - DailyCI

In a main branch, we trigger daily builds on a daily basis, and complete the ST system (including stress tester, etc.) test. The execution of the same I will generate the results of the message and push it to the team members. Due to resource constraints, we set the scheduled time to start in the middle of the night, but we can see the results on the second day of work and make full use of automation to fill in the free time for everyone to rest in the evening.

In addition, the relevant thresholds can be set during the scanning process, and the red lights that do not meet the requirements can be turned on for rectification.

(Note: The CI principle is: The red light does not stay overnight. If it cannot be repaired in time, it can be returned first, and then re-entered after being resolved.)

8. Multi-dimensional product library management

From the product library type: A product library is divided into three libraries: snapshot, alpha, and release. The functions of the three libraries are different. The snapshot library stores the daily build version of the CI. The alpha library stores the stable version for internal testing, and the release stores the external release version.

1) snapshot product library

What is stored is the daily CI build process version. This type of version passes basic automated tests and is generally used for subsequent temporary problem verification and mainline iteration manual testing.

2) Alpha product library

Stored is a stable and iterative version that has been verified by automation and manual testing. This type of version is mainly for internal self-testing and can be released internally, and submitted to the test department for a system test that accounts for about 20% of the subsequent.

3) release product library

Stored through the system test, you can actually release the deployed version.

4) Grayscale publishing & continuous optimization

Release version of the product library, planners will promote the deployment of small-scale grayscale upgrades in the field, tracking the changes in the operation and maintenance indicators after the upgrade, if no abnormalities are found then arrange for a large-scale upgrade and deployment; if there is an abnormal need to repair The feedback was re-iterated and released by the development team to form a PDCA closed loop from requirements to development testing and operation and maintenance.

Practice

Through the end-to-end process transformation of the project, the development process of a probe project needs to be smoother, the quality of the team's output is significantly improved, and the timely and benign interactions among the needs, development, testing, and operation and maintenance personnel are formed.

Effect evaluation :

Branch code from svn to git, download speed, improve version build efficiency

Demand has been fully discussed and reviewed, and split in different dimensions

Realize the task management visualization, can timely intervene or adjust the risk of priority management and control

The code submission, review to the main branch storage for effective control

There are timely and effective feedback loops for review, CI testing, and static inspection

Using DevOps Practice Comparison

	past	just now	income
Requirements task visualization	Sudden demand, lack of communication with actual developers, unclear priority in coordination when tasks conflict	Presented by Feature, User Story, Task, and Bug. Demand description and acceptance criteria. Decompose demand before clarification.	The iterative output version is more in line with the market delivery requirements, and the version quality, development efficiency, and team cooperation are improved.
Version build	Download takes 5~10min, no network can't work	Download is not time consuming	Version build basically does not spend download time, disconnected network can also work
Check execution time	After the code is put into the library	Code submission action triggers immediate execution	Find problems and give feedback in 5 minutes
automated test	Less test cases, difficult integration of test tools, difficulty in expansion, and inconsistent check results	Unified RF-based automated test design	The test cases are platform-based, easy to supplement and expand, and the results are unified in the push format.
Coverage check	no	Grov/lcov performs detailed output report for reference improvement	Guarantee coverage, guarantee code quality, improve test case coverage
Static inspection	no	Check the change code before warehousing, all requirements are met	Improve code quality
Operation and maintenance improvement	Passive improvement after version release and deployment	Post-deployment pilot observations, continuous improvement based on data	Formed a closed loop of requirements, development, testing and operation and maintenance

Promotion

This article briefly describes the process of a project's attempt to implement the DevOps tool chain and compares the effects before and after it. It explores end-to-end delivery and optimizes the R&D process to improve the overall efficiency of the team. Of course, in many ways we have not done enough. Need to continue to optimize and improve, this transformation process is worthy of reference for related projects.