gen_tcp接受链接时enfile的问题分析及解决

    最近我们为了安全方面的原因，在RDS服务器上做了个代理程序把普通的MYSQL TCP连接变成了SSL链接，在测试的时候，皓庭同学发现Tsung发起了几千个TCP链接后Erlang做的SSL PROXY老是报告gen_tcp:accept返回{error, enfile}错误。针对这个问题，我展开了如下的调查：

    首先man accept手册，确定enfile的原因，因为gen_tcp肯定是调用accept系统调用的:

     EMFILE The per-process limit of open file descriptors has been reached.

     ENFILE The system limit on the total number of open files has been reached.

    从文档来看是由于系统的文件句柄数用完了，我们顺着来调查下：

$ uname -r
2.6.18-164.el5
$ cat /proc/sys/fs/file-nr
2040    0       2417338
$ ulimit -n
65535

    由于我们微调了系统的文件句柄，具体参考这里 老生常谈: ulimit问题及其影响， 这些参数看起来非常的正常。

     先看下net/socket.c代码：

static int sock_alloc_fd(struct file **filep)
{
        int fd;

        fd = get_unused_fd();
        if (likely(fd >= 0)) {
                struct file *file = get_empty_filp();

                *filep = file;
                if (unlikely(!file)) {
                        put_unused_fd(fd);
                        return -ENFILE;
                }
        } else
                *filep = NULL;
        return fd;
}

static int __sock_create(int family, int type, int protocol, struct socket **res, int kern)
{
...
/*
 *      Allocate the socket and allow the family to set things up. if
 *      the protocol is 0, the family is instructed to select an appropriate
 *      default.
 */

        if (!(sock = sock_alloc())) {
                if (net_ratelimit())
                        printk(KERN_WARNING "socket: no more sockets\\n");
                err = -ENFILE;          /* Not exactly a match, but its the
                                           closest posix thing */
                goto out;
        }
...
}

asmlinkage long sys_accept(int fd, struct sockaddr __user *upeer_sockaddr, int __user *upeer_addrlen)
{
        struct socket *sock, *newsock;
        struct file *newfile;
        int err, len, newfd, fput_needed;
        char address[MAX_SOCK_ADDR];

        sock = sockfd_lookup_light(fd, &err, &fput_needed);
        if (!sock)
                goto out;

        err = -ENFILE;
        if (!(newsock = sock_alloc()))
                goto out_put;
...
}

    从代码来看，会返回ENFILE都是由于socket句柄分配不出来了，我们还是本着怀疑的态度来写个stap脚本来再次验证下：

$ cat enfile.stp
probe kernel.function("kmem_cache_alloc").return,
          kernel.function("get_empty_filp").return{
  if($return == 0) { print_backtrace();exit();}
}
probe kernel.function("sock_alloc_fd").return {
  if($return < 0) { print_backtrace(); exit();}
}
probe syscall.accept.return {
  if($return == -23) {print_backtrace(); exit();}
}
probe begin {
println(":~");
}
$ sudo stap enfile.stp
:~

    gen_tcp:accept报告{error, enfile}的时候，也没看到stap报异常，基本上可以排除操作系统的原因了，那么我们现在回到gen_tcp的实现来看。

     gen_tcp是个port, 具体实现在erts/emulator/drivers/common/inet_drv.c，我们来看下有ENFILE的地方：

/* Copy a descriptor, by creating a new port with same settings
 * as the descriptor desc.
 * return NULL on error (ENFILE no ports avail)
 */
static tcp_descriptor* tcp_inet_copy(tcp_descriptor* desc,SOCKET s,
                                     ErlDrvTermData owner, int* err)
{
...
    /* The new port will be linked and connected to the original caller */
    port = driver_create_port(port, owner, "tcp_inet", (ErlDrvData) copy_desc);
    if ((long)port == -1) {
        *err = ENFILE;
        FREE(copy_desc);
        return NULL;
    }
...
}

    当 driver_create_port 失败的时候，gen_tcp返回ENFILE,看起来这次找对地方了。我们继续看下 driver_create_port的实现：

     看下erts/emulator/beam/io.c：

/*
 * Driver function to create new instances of a driver
 * Historical reason: to be used with inet_drv for creating
 * accept sockets inorder to avoid a global table.
 */
ErlDrvPort
driver_create_port(ErlDrvPort creator_port_ix, /* Creating port */
                   ErlDrvTermData pid,    /* Owner/Caller */
                   char* name,            /* Driver name */
                   ErlDrvData drv_data)   /* Driver data */
{
...
    rp = erts_pid2proc(NULL, 0, pid, ERTS_PROC_LOCK_LINK);
    if (!rp) {
        erts_smp_mtx_unlock(&erts_driver_list_lock);
        return (ErlDrvTermData) -1;   /* pid does not exist */
    }
    if ((port_num = get_free_port()) < 0) {
        errno = ENFILE;
        erts_smp_proc_unlock(rp, ERTS_PROC_LOCK_LINK);
        erts_smp_mtx_unlock(&erts_driver_list_lock);
        return (ErlDrvTermData) -1;
    }

    port_id = make_internal_port(port_num);
    port = &erts_port[port_num & erts_port_tab_index_mask];
...
}

    get_free_port() Basic Applications -> erts -> erl -> System Flags:

     +K true|false

     Enables or disables the kernel poll functionality if the emulator has kernel poll support. By default the kernel poll; functionality is disabled. If the emulator doesn\'t have kernel poll support and the +K flag is passed to the emulator, a warning is issued at startup.

     If you meet all requirements, you can enable it in this way:

     erl -s ejabberd +K true ...

    Mnesia Tables to Disk

     By default, ejabberd uses Mnesia as its database. In Mnesia you can configure each table in the database to be stored on RAM, on RAM and on disk, or only on disk. You can configure this in the web interface: Nodes -> \'mynode\' -> DB Management. Modification of this option will consume some memory and CPU time.

     Number of Concurrent ETS and Mnesia Tables: ERL_MAX_ETS_TABLES

     The number of concurrent ETS and Mnesia tables is limited. When the limit is reached, errors will appear in the logs:

     ** Too many db tables **

     You can safely increase this limit when starting ejabberd. It impacts memory consumption but the difference will be quite small.

     erl -s ejabberd -env ERL_MAX_ETS_TABLES 20000 ...

    小结：很多问题是很绕的，要多方面考虑验证。