Millet Porridge

English version of https://corvo.myseu.cn

0%

Reading the Nginx Reload Process Source Code

Posted on Edited on In Disqus:
I was recently considering using Nginx's keepalive parameter and originally wanted to explore how Nginx implements keepalive, but ended up mainly studying the reload process — quite a detour. Just the right chance to share this part of the code with everyone.

This assumes readers have some knowledge of Linux system calls — at least a simple understanding of the fork operation and simple inter-process communication.

A brief exploration of the master-slave communication mechanism and process.

Entering the Reload Logic

Normally I reload Nginx on servers with roughly these two operations:

1
2
nginx -s reload
kill -HUP `cat /var/run/nginx.pid`

Reloading Nginx means sending the SIGHUP signal to Nginx’s master process; the master then creates new workers and removes old workers.

Signal sending and handling isn’t this blog’s focus; if you’re interested in that process, see here:

Understanding nginx -s reload from the nginx 1.17.9 source

I’ll start directly from where the reload is checked:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
// src/os/unix/ngx_process_cycle.c
void
ngx_master_process_cycle(ngx_cycle_t *cycle)
{

    for(;;) {
        if (ngx_reconfigure) {
            ngx_reconfigure = 0;
            fprintf(stderr, "[ngx_master_process_cycle] reconfigure\n");

            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reconfiguring");

            cycle = ngx_init_cycle(cycle);
            if (cycle == NULL) {
                cycle = (ngx_cycle_t *) ngx_cycle;
                continue;
            }

            ngx_cycle = cycle;
            ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx,
                                                   ngx_core_module);

            fprintf(stderr, "[ngx_master_process_cycle] start new worker process \n");
            // start new worker processes
            ngx_start_worker_processes(cycle, ccf->worker_processes,
                                       NGX_PROCESS_JUST_RESPAWN);
            ngx_start_cache_manager_processes(cycle, 1);

            /* allow new processes to start */
            ngx_msleep(100);

            live = 1;

            fprintf(stderr, "[ngx_master_process_cycle] shutdown old worker process\n");
            // shut down old worker processes
            ngx_signal_worker_processes(cycle,
                                        ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
        }
    }
}

Creating New Worker Processes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
static void
ngx_start_worker_processes(ngx_cycle_t *cycle, ngx_int_t n, ngx_int_t type)
{
    for (i = 0; i < n; i++) {
        // start the Nginx process and call `ngx_worker_process_cycle` in the new process
        ngx_spawn_process(cycle, ngx_worker_process_cycle,
                          (void *) (intptr_t) i, "worker process", type);

        ch.pid = ngx_processes[ngx_process_slot].pid;
        ch.slot = ngx_process_slot;
        ch.fd = ngx_processes[ngx_process_slot].channel[0];

        ngx_pass_open_channel(cycle, &ch);
    }
}

Nginx’s Process Startup

The process-starting function is fairly complex; I removed some logic unused during reload. Let me explain the parts I understand.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
// src/os/unix/ngx_process.c
ngx_pid_t
ngx_spawn_process(ngx_cycle_t *cycle, ngx_spawn_proc_pt proc, void *data,
    char *name, ngx_int_t respawn)
{
    u_long     on;
    ngx_pid_t  pid;
    ngx_int_t  s;
  
        // ngx_processes stores all of Nginx's process variables;
        // the logic below: find the first available process_t struct and break out of the loop once found
        for (s = 0; s < ngx_last_process; s++) {
            if (ngx_processes[s].pid == -1) {
                break;
            }
        }

    if (respawn != NGX_PROCESS_DETACHED) {

        /* Solaris 9 still has no AF_LOCAL */

        // create a channel for master-slave communication. Personally I dislike this function's control flow;
        // also the code below suits being written as a separate function using goto statements —
        // avoiding multiple returns and guaranteeing cleanup on error. I'll give my considered pseudocode in the appendix
        if (socketpair(AF_UNIX, SOCK_STREAM, 0, ngx_processes[s].channel) == -1)
        {
            ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                          "socketpair() failed while spawning \"%s\"", name);
            return NGX_INVALID_PID;
        }

        // handling the newly created socket. I don't normally write socket code and don't know these parameters well;
        // for what each line does, see: https://zhuanlan.zhihu.com/p/96757160

        // very important: ngx_channel here is a global variable; the child process copies the value over at fork
        ngx_channel = ngx_processes[s].channel[1];
    }

    ngx_process_slot = s;

    pid = fork();

    switch (pid) {

    case -1:
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "fork() failed while spawning \"%s\"", name);
        ngx_close_channel(ngx_processes[s].channel, cycle->log);
        return NGX_INVALID_PID;

    case 0:
        // the created child process will execute `ngx_worker_process_cycle`
        ngx_parent = ngx_pid;
        ngx_pid = ngx_getpid();
        proc(cycle, data);
        break;

    default:
        break;
    }

    ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "start %s %P", name, pid);

    // below are some state-sync operations I don't fully understand; variables in ngx_processes are synced according to the respawn parameter

    return pid;
}

Worker Process Initialization and State Handling

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
static void
ngx_worker_process_cycle(ngx_cycle_t *cycle, void *data)
{
    ngx_int_t worker = (intptr_t) data;

    ngx_process = NGX_PROCESS_WORKER;
    ngx_worker = worker;

    // the worker process's initialization operations
    ngx_worker_process_init(cycle, worker);

    ngx_setproctitle("worker process");

    for ( ;; ) {

        if (ngx_exiting) {
            if (ngx_event_no_timers_left() == NGX_OK) {
                ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");
                ngx_worker_process_exit(cycle);
            }
        }

        ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "worker cycle");

        ngx_process_events_and_timers(cycle);

        if (ngx_terminate) {
            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");
            ngx_worker_process_exit(cycle);
        }

        if (ngx_quit) {
            ngx_quit = 0;
            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0,
                          "gracefully shutting down");
            ngx_setproctitle("worker process is shutting down");

            if (!ngx_exiting) {
                ngx_exiting = 1;
                ngx_set_shutdown_timer(cycle);
                ngx_close_listening_sockets(cycle);
                ngx_close_idle_connections(cycle);
            }
        }

        if (ngx_reopen) {
            ngx_reopen = 0;
            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reopening logs");
            ngx_reopen_files(cycle, -1);
        }
    }
}


static void
ngx_worker_process_init(ngx_cycle_t *cycle, ngx_int_t worker)
{
    sigset_t          set;
    ngx_int_t         n;
    ngx_time_t       *tp;
    ngx_uint_t        i;
    ngx_cpuset_t     *cpu_affinity;
    struct rlimit     rlmt;
    ngx_core_conf_t  *ccf;
    ngx_listening_t  *ls;

    if (ngx_set_environment(cycle, NULL) == NULL) {
        /* fatal */
        exit(2);
    }

    ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module);

    // set priority
    // set limits
    // handle linux user/group issues
    // set cpu affinity
    // change directory

    // initialize Nginx's modules
    for (i = 0; cycle->modules[i]; i++) {
        if (cycle->modules[i]->init_process) {
            if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {
                exit(2);
            }
        }
    }

    // close ch[1] of other child processes besides the current one,
    // because the ngx_processes struct was also copied at fork,
    // and the current child process doesn't need to access other processes' channels
    for (n = 0; n < ngx_last_process; n++) {

        if (ngx_processes[n].pid == -1) {
            continue;
        }

        if (n == ngx_process_slot) {
            continue;
        }

        if (ngx_processes[n].channel[1] == -1) {
            continue;
        }

        if (close(ngx_processes[n].channel[1]) == -1) {
            ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                          "close() channel failed");
        }
    }

    // close the current ch[0]; the child process keeps only ch[1]
    if (close(ngx_processes[ngx_process_slot].channel[0]) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "close() channel failed");
    }

    // add listener functions; when different signals are received, the ngx_quit, ngx_terminate, ngx_reopen variables are modified
    if (ngx_add_channel_event(cycle, ngx_channel, NGX_READ_EVENT,
                              ngx_channel_handler)
        == NGX_ERROR)
    {
        /* fatal */
        exit(2);
    }
}

The Parent-Child Communication Process

In Nginx, process communication is based on socketpair, with parent and child each holding one socket. For the Nginx process creation part, I’ll write brief pseudocode, hoping to help understanding:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
To create the i-th process

socketpair initializes ngx_processes[i].channel, ch[0,1]


// ngx_channel here is a global variable; the child copies the value at fork
// personally I think this operation could also be done in the child process; those with questions can speak up
ngx_channel = ngx_processes[i].channel[1];

pid = fork()

if pid == 0 {
    # child process
    keep only the current process's ch[1] for communicating with the parent
    i.e. communicate using ngx_channel
else {
    # parent process
    ngx_processes[i].channel[0] is used to communicate with the i-th child process
}

Deleting Old Workers

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
static void
ngx_signal_worker_processes(ngx_cycle_t *cycle, int signo)
{
    ngx_int_t      i;
    ngx_err_t      err;
    ngx_channel_t  ch;

    ngx_memzero(&ch, sizeof(ngx_channel_t));

    switch (signo) {

    case ngx_signal_value(NGX_SHUTDOWN_SIGNAL):
        ch.command = NGX_CMD_QUIT;
        break;

    case ngx_signal_value(NGX_TERMINATE_SIGNAL):
        ch.command = NGX_CMD_TERMINATE;
        break;

    case ngx_signal_value(NGX_REOPEN_SIGNAL):
        ch.command = NGX_CMD_REOPEN;
        break;

    default:
        ch.command = 0;
    }

    ch.fd = -1;


    for (i = 0; i < ngx_last_process; i++) {

        if (ngx_processes[i].detached || ngx_processes[i].pid == -1) {
            continue;
        }

        // a just-created process is in just_spawn state, so newly created processes enter the if statement and are not processed further
        if (ngx_processes[i].just_spawn) {
            ngx_processes[i].just_spawn = 0;
            continue;
        }

        // exiting processes are also not processed
        if (ngx_processes[i].exiting
            && signo == ngx_signal_value(NGX_SHUTDOWN_SIGNAL))
        {
            continue;
        }

        // send a signal to the child process; the parameter given at destruction is SHUTDOWN, so the command is NGX_CMD_QUIT
        if (ch.command) {
            fprintf(stderr, "[ngx_signal_worker_processes] write channel (%d, %d, %ld)\n", ngx_processes[i].pid, signo, ch.command);
            if (ngx_write_channel(ngx_processes[i].channel[0],
                                  &ch, sizeof(ngx_channel_t), cycle->log)
                == NGX_OK)
            {
                if (signo != ngx_signal_value(NGX_REOPEN_SIGNAL)) {
                    ngx_processes[i].exiting = 1;
                }

                continue;
            }
        }

        ngx_log_debug2(NGX_LOG_DEBUG_CORE, cycle->log, 0,
                       "kill (%P, %d)", ngx_processes[i].pid, signo);

        if (kill(ngx_processes[i].pid, signo) == -1) {
            err = ngx_errno;
            ngx_log_error(NGX_LOG_ALERT, cycle->log, err,
                          "kill(%P, %d) failed", ngx_processes[i].pid, signo);

            if (err == NGX_ESRCH) {
                ngx_processes[i].exited = 1;
                ngx_processes[i].exiting = 0;
                ngx_reap = 1;
            }

            continue;
        }

        if (signo != ngx_signal_value(NGX_REOPEN_SIGNAL)) {
            ngx_processes[i].exiting = 1;
        }
    }
}

Summary

References

How nginx master-worker processes work nginx source analysis 1 — inter-process communication mechanisms (semaphores)

Appendix

Thoughts on Code Optimization After socketpair Initialization

The original code is like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
// src/os/unix/ngx_process.c
if (socketpair(AF_UNIX, SOCK_STREAM, 0, ngx_processes[s].channel) == -1)
{
    ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                  "socketpair() failed while spawning \"%s\"", name);
    return NGX_INVALID_PID;
}

ngx_log_debug2(NGX_LOG_DEBUG_CORE, cycle->log, 0,
               "channel %d:%d",
               ngx_processes[s].channel[0],
               ngx_processes[s].channel[1]);

if (ngx_nonblocking(ngx_processes[s].channel[0]) == -1) {
    ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                  ngx_nonblocking_n " failed while spawning \"%s\"",
                  name);
    ngx_close_channel(ngx_processes[s].channel, cycle->log);
    return NGX_INVALID_PID;
}

if (ngx_nonblocking(ngx_processes[s].channel[1]) == -1) {
    ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                  ngx_nonblocking_n " failed while spawning \"%s\"",
                  name);
    ngx_close_channel(ngx_processes[s].channel, cycle->log);
    return NGX_INVALID_PID;
}

on = 1;
if (ioctl(ngx_processes[s].channel[0], FIOASYNC, &on) == -1) {
    ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                  "ioctl(FIOASYNC) failed while spawning \"%s\"", name);
    ngx_close_channel(ngx_processes[s].channel, cycle->log);
    return NGX_INVALID_PID;
}

if (fcntl(ngx_processes[s].channel[0], F_SETOWN, ngx_pid) == -1) {
    ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                  "fcntl(F_SETOWN) failed while spawning \"%s\"", name);
    ngx_close_channel(ngx_processes[s].channel, cycle->log);
    return NGX_INVALID_PID;
}

if (fcntl(ngx_processes[s].channel[0], F_SETFD, FD_CLOEXEC) == -1) {
    ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                  "fcntl(FD_CLOEXEC) failed while spawning \"%s\"",
                   name);
    ngx_close_channel(ngx_processes[s].channel, cycle->log);
    return NGX_INVALID_PID;
}

if (fcntl(ngx_processes[s].channel[1], F_SETFD, FD_CLOEXEC) == -1) {
    ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                  "fcntl(FD_CLOEXEC) failed while spawning \"%s\"",
                   name);
    ngx_close_channel(ngx_processes[s].channel, cycle->log);
    return NGX_INVALID_PID;
}

You can see the return statement is used multiple times, and ngx_close_channel is also used multiple times. Having looked at Linux kernel code before, the code above can actually be simplified with goto statements. Below is my simplified version:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
// C syntax is quite detail-oriented; I'll just gloss over it with pseudocode — maybe submit a PR someday
init_channel() {
    if (socketpair(AF_UNIX, SOCK_STREAM, 0, ngx_processes[s].channel) == -1)
    {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "socketpair() failed while spawning \"%s\"", name);
        goto error;
    }

    if (ngx_nonblocking(ngx_processes[s].channel[0]) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      ngx_nonblocking_n " failed while spawning \"%s\"",
                      name);
        goto error;
    }

    if (ngx_nonblocking(ngx_processes[s].channel[1]) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      ngx_nonblocking_n " failed while spawning \"%s\"",
                      name);
        goto error;
    }

    on = 1;
    if (ioctl(ngx_processes[s].channel[0], FIOASYNC, &on) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "ioctl(FIOASYNC) failed while spawning \"%s\"", name);
        goto error;
    }

    if (fcntl(ngx_processes[s].channel[0], F_SETOWN, ngx_pid) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "fcntl(F_SETOWN) failed while spawning \"%s\"", name);
        goto error;
    }

    if (fcntl(ngx_processes[s].channel[0], F_SETFD, FD_CLOEXEC) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "fcntl(FD_CLOEXEC) failed while spawning \"%s\"",
                       name);
        goto error;
    }

    if (fcntl(ngx_processes[s].channel[1], F_SETFD, FD_CLOEXEC) == -1) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                      "fcntl(FD_CLOEXEC) failed while spawning \"%s\"",
                       name);
        goto error;
    }

    return NGX_OK;

error:
    if (channel != -1) {
        ngx_close_channel(ngx_processes[s].channel, cycle->log);
    }
    return NGX_INVALID_PID;
}

Adding goto statements here actually simplifies much of the logic, and you needn’t worry about forgetting close_channel after errors — this feels like an appropriate place to use goto.

How to Print Your Own Logs in Nginx

I couldn’t figure out why calling ngx_log_error myself printed no logs, so I just used the fprintf(stderr, "hello world"); form directly — it’s only debugging anyway.

https://stackoverflow.com/questions/20187630/nginx-logging-in-module

Related Posts