Net | GNUers 博客

checksum error的原因

今天有同事反馈dig @223.5.5.5的时候看到本地发出去的包是提示“bad udp cksum” xxx > 223.5.5.5.53: [bad udp cksum 0x85e1 -> 0xc2e3!] 8250+ A? www.baidu.com. 实际这个是因为网卡开启了tx checksum，开启之后这个checksum的计算是由网卡硬件自己完成，tcpdump抓包的时候实际还没有去结算checksum，所以一直是bad upd cksum #ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: off tx-vlan-offload: off ntuple-filters: off receive-hashing: off 只能在目标机器进行抓包才能发现是否发出的包checkum是否真的有错误。另外可以选择本地把tx checksum关闭掉 #ethtool -K eth1 tx off 再测试的时候可以看到是OK的了 xxx. > 223.5.5.5.53: [udp sum ok] 44024+ A? www.baidu.com. (31) 实际利用网卡计算checksum显然更好，所以不用太在意这个。

arp_announce引发的1个ARP问题

最近配置服务器时遇到在dummy0上宣告的公网地址不能正常访问公网的问题。网络的基本结构为2个网卡分别上联2个交换机，跑OSPF。在dummy0上会单独宣告1个公网的地址。自己通过在zebra内修改路由表使得访问公网时设置自己宣告的公网地址为源IP。以前这个方案实际线上跑了多次，一直OK。最近有一套服务器安装这样的配置会出现刚启动时是OK的，过段时间就歇菜了。简单地说实际的情况如下：T1 (192.168.1.2 )–> 交换机A的Port X(192.168.1.1)T2 (192.168.2.2 )–> 交换机B的Port X(192.168.2.1)平时默认都走T1，公网的路由表默认学到的网关实际是T2的对端地址。当本地公网地址不能出去的时候，我自己带源地址(架设公网地址是4.4.4.4)ping，发现发送的ARP请求都是这样 Request who-has 192.168.2.1 tell 4.4.4.4 此时交换机上发现4.4.4.4实际不是和自己一个网段的地址，不会进行回复。就使得这本地这个公网地址出不去。解决这个问题就是加内核参数 net.ipv4.conf.all.arp_announce=1 下面是fix后的情况，可以看到使用的是接口上的IP为源IP发的ARP请求。附上参数说明 rp_announce - INTEGER Define different restriction levels for announcing the local source IP address from IP packets in ARP requests sent on interface: 0 - (default) Use any local address, configured on any interface 1 - Try to avoid local addresses that are not in the target's subnet for this interface. This mode is useful when target ...

配置OpenVPN只注入特定路由表

通过情况下大家配置VPN时，都会直接把默认网关指向服务端。但是这会影响访问公司内网资源，造成一些不便。如果只是希望走VPN访问部分外网资源，可以不让VPN客户端改默认的网关，通过加一些路由表使得到特定地址走VPN。比如我在公司的时候只是需要访问一下google，可以在VPN服务端新增配置,其中把push redirect-gateway def1 bypass-dhcp给直接注释掉，新push了很多路由表到客户端。另外，在客户端也需要把redirect-gateway给注释掉。这样启动后可以看到只是注入特定的路由表，没有改默认路由。附上服务端配置： port 600xxxproto udpdev tunca key/ca.crtcert key/server.crtkey key/server.key # This file should be kept secretdh key/dh1024.pemserver 10.99.1.0 255.255.255.0ifconfig-pool-persist ipp.txtpush "dhcp-option DNS 208.67.220.220"push "dhcp-option DNS 208.67.222.222"#push redirect-gateway def1 bypass-dhcpkeepalive 10 30comp-lzomax-clients 60user nobodygroup nogrouppersist-keypersist-tunstatus openvpn-google.logverb 3mute 20duplicate-cn add google route rulepush “route 207.223.160.0 255.255.240.0 “push “route 66.249.85.0 255.255.255.0 “push “route 66.249.83.0 255.255.255.0 “push “route 74.125.130.0 255.255.255.0 “push “route 192.178.0.0 255.254.0.0 “push “route 64.233.160.0 255.255.255.0 “push “route 74.125.17.0 255.255.255.0 “push “route 66.249.72.0 255.255.255.0 “push “route 173.194.112.0 255.255.255.0 “push “route 173.194.98.0 255.255.255.0 “push “route 173.194.140.0 255.255.255.0 “push “route 74.125.196.0 255.255.255.0 “push “route 173.194.78.0 255.255.255.0 “push “route 209.85.238.0 255.255.255.0 “push “route 72.14.208.0 255.255.254.0 “push “route 64.233.164.0 255.255.255.0 “push “route 8.15.202.0 255.255.255.0 “push “route 74.125.142.0 255.255.255.0 “push “route 108.177.0.0 255.255.128.0 “push “route 74.125.203.0 255.255.255.0 “push “route 74.125.58.0 255.255.255.0 “push “route 173.194.141.0 255.255.255.0 “push “route 72.14.244.0 255.255.254.0 “push “route 173.194.73.0 255.255.255.0 “push “route 72.14.225.0 255.255.255.0 “push “route 74.125.193.0 255.255.255.0 “push “route 74.125.239.0 255.255.255.0 “push “route 173.255.112.0 255.255.240.0 “push “route 173.194.119.0 255.255.255.0 “push “route 66.249.64.0 255.255.224.0 “push “route 66.249.70.0 255.255.255.0 “push “route 74.125.190.0 255.255.255.0 “push “route 74.125.70.0 255.255.255.0 “push “route 74.125.206.0 255.255.255.0 “push “route 74.125.198.0 255.255.255.0 “push “route 173.194.75.0 255.255.255.0 “push “route 8.34.208.0 255.255.248.0 “push “route 74.125.19.0 255.255.255.0 “push “route 74.125.131.0 255.255.255.0 “push “route 66.102.4.0 255.255.255.0 “push “route 173.194.76.0 255.255.255.0 “push “route 8.34.216.0 255.255.248.0 “push “route 66.249.79.0 255.255.255.0 “push “route 66.249.90.0 255.255.255.0 “push “route 162.216.148.0 255.255.252.0 “push “route 173.194.32.0 255.255.255.0 “push “route 173.194.142.0 255.255.255.0 “push “route 74.125.238.0 255.255.255.0 “push “route 74.125.18.0 255.255.255.0 “push “route 74.125.234.0 255.255.255.0 “push “route 173.194.79.0 255.255.255.0 “push “route 173.194.40.0 255.255.255.0 “push “route 66.249.64.0 255.255.255.0 “push “route 74.125.68.0 255.255.255.0 “push “route 74.125.43.0 255.255.255.0 “push “route 192.158.28.0 255.255.252.0 “push “route 8.35.192.0 255.255.248.0 “push “route 74.125.0.0 255.255.0.0 “push “route 209.85.128.0 255.255.128.0 “push “route 66.249.67.0 255.255.255.0 “push “route 66.249.84.0 255.255.255.0 “push “route 1.2.3.0 255.255.255.0 “push “route 74.125.232.0 255.255.255.0 “push “route 173.194.96.0 255.255.255.0 “push “route 74.125.118.0 255.255.255.0 “push “route 74.125.28.0 255.255.255.0 “push “route 173.194.121.0 255.255.255.0 “push “route 70.32.144.0 255.255.255.0 “push “route 74.125.186.0 255.255.255.0 “push “route 74.125.31.0 255.255.255.0 “push “route 64.233.166.0 255.255.255.0 “push “route 74.125.207.0 255.255.255.0 “push “route 8.8.8.0 255.255.255.0 “push “route 173.194.65.0 255.255.255.0 “push “route 74.125.138.0 255.255.255.0 “push “route 173.194.34.0 255.255.255.0 “push “route 74.125.192.0 255.255.255.0 “push “route 66.249.91.0 255.255.255.0 “push “route 74.125.229.0 255.255.255.0 “push “route 74.125.88.0 255.255.254.0 “push “route 74.125.37.0 255.255.255.0 “push “route 74.125.40.0 255.255.255.0 “push “route 74.125.176.0 255.255.255.0 “push “route 64.233.171.0 255.255.255.0 “push “route 173.194.70.0 255.255.255.0 “push “route 193.142.125.0 255.255.255.0 “push “route 74.125.187.0 255.255.255.0 “push “route 74.125.29.0 255.255.255.0 “push “route 74.125.16.0 255.255.255.0 “push “route 66.249.65.0 255.255.255.0 “push “route 173.194.66.0 255.255.255.0 “push “route 74.125.202.0 255.255.255.0 “push “route 173.194.68.0 255.255.255.0 “push “route 173.194.120.0 255.255.255.0 “push “route 173.194.113.0 255.255.255.0 “push “route 216.239.38.0 255.255.255.0 “push “route 146.148.0.0 255.255.128.0 “push “route 64.233.160.0 255.255.224.0 “push “route 66.102.2.0 255.255.255.0 “push “route 66.249.88.0 255.255.255.0 “push “route 72.14.192.0 255.255.192.0 “push “route 66.249.78.0 255.255.255.0 “push “route 173.194.45.0 255.255.255.0 “push “route 74.125.183.0 255.255.255.0 “push “route 74.125.230.0 255.255.255.0 “push “route 74.125.129.0 255.255.255.0 “push “route 70.32.148.0 255.255.254.0 “push “route 172.253.0.0 255.255.0.0 “push “route 74.125.116.0 255.255.255.0 “push “route 173.194.97.0 255.255.255.0 “push “route 64.233.186.0 255.255.255.0 “push “route 23.236.48.0 255.255.240.0 “push “route 74.125.76.0 255.255.255.0 “push “route 74.125.26.0 255.255.255.0 “push “route 74.125.36.0 255.255.255.0 “push “route 74.125.63.0 255.255.255.0 “push “route 66.249.80.0 255.255.255.0 “push “route 142.250.0.0 255.254.0.0 “push “route 173.194.124.0 255.255.255.0 “push “route 64.233.172.0 255.255.255.0 “push “route 173.194.43.0 255.255.255.0 “push “route 23.251.128.0 255.255.224.0 “push “route 74.125.235.0 255.255.255.0 “push “route 162.222.176.0 255.255.248.0 “push “route 74.125.137.0 255.255.255.0 “push “route 173.194.72.0 255.255.255.0 “push “route 173.194.136.0 255.255.255.0 “push “route 74.125.188.0 255.255.255.0 “push “route 74.125.185.0 255.255.255.0 “push “route 173.194.42.0 255.255.255.0 “push “route 74.125.226.0 255.255.255.0 “push “route 74.125.227.0 255.255.255.0 “push “route 216.239.35.0 255.255.255.0 “push “route 1.0.0.0 255.255.255.0 “push “route 173.194.46.0 255.255.255.0 “push “route 74.125.205.0 255.255.255.0 “push “route 216.239.34.0 255.255.255.0 “push “route 74.125.117.0 255.255.255.0 “push “route 173.194.44.0 255.255.255.0 “push “route 74.125.182.0 255.255.255.0 “push “route 74.125.178.0 255.255.255.0 “push “route 74.125.30.0 255.255.255.0 “push “route 216.239.39.0 255.255.255.0 “push “route 74.125.231.0 255.255.255.0 “push “route 66.249.92.0 255.255.255.0 “push “route 66.102.0.0 255.255.240.0 “push “route 216.239.44.0 255.255.254.0 “push “route 74.125.74.0 255.255.255.0 “push “route 173.194.33.0 255.255.255.0 “push “route 216.58.192.0 255.255.224.0 “push “route 173.194.117.0 255.255.255.0 “push “route 74.125.191.0 255.255.255.0 “push “route 74.125.22.0 255.255.255.0 “push “route 173.194.35.0 255.255.255.0 “push “route 74.125.201.0 255.255.255.0 “push “route 216.239.32.0 255.255.224.0 “push “route 74.125.121.0 255.255.255.0 “push “route 66.249.89.0 255.255.255.0 “push “route 108.59.80.0 255.255.240.0 “push “route 74.125.224.0 255.255.255.0 “push “route 172.217.0.0 255.255.0.0 “push “route 74.125.119.0 255.255.255.0 “push “route 113.197.106.0 255.255.255.0 “push “route 64.233.173.0 255.255.255.0 “push “route 66.102.3.0 255.255.255.0 “push “route 74.125.177.0 255.255.255.0 “push “route 74.125.41.0 255.255.255.0 “push “route 74.125.189.0 255.255.255.0 “push “route 74.125.24.0 255.255.255.0 “push “route 74.125.236.0 255.255.255.0 “push “route 74.125.143.0 255.255.255.0 “push “route 8.35.200.0 255.255.248.0 “push “route 173.194.67.0 255.255.255.0 “push “route 72.14.228.0 255.255.255.0 “push “route 173.194.36.0 255.255.255.0 “push “route 74.125.184.0 255.255.255.0 “push “route 64.233.168.0 255.255.255.0 “push “route 173.194.41.0 255.255.255.0 “push “route 74.125.90.0 255.255.254.0 “push “route 173.194.118.0 255.255.255.0 “push “route 173.194.37.0 255.255.255.0 “push “route 107.178.192.0 255.255.192.0 “push “route 173.194.0.0 255.255.0.0 “push “route 74.125.180.0 255.255.255.0 “push “route 74.125.200.0 255.255.255.0 “push “route 74.125.233.0 255.255.255.0 “push “route 74.125.122.0 255.255.255.0 “push “route 70.32.128.0 255.255.224.0 “push “route 130.211.0.0 255.255.0.0 “push “route 74.125.237.0 255.255.255.0 “push “route 74.125.42.0 255.255.255.0 “push “route 173.194.64.0 255.255.255.0 “push “route 74.125.20.0 255.255.255.0 “push “route 173.194.127.0 255.255.255.0 “push “route 74.125.128.0 255.255.255.0 “push “route 173.194.39.0 255.255.255.0 “push “route 74.125.194.0 255.255.255.0 “push “route 66.249.77.0 255.255.255.0 “push “route 173.194.69.0 255.255.255.0 “push “route 74.125.136.0 255.255.255.0 “push “route 74.125.54.0 255.255.254.0 “push “route 173.194.99.0 255.255.255.0 “push “route 66.249.74.0 255.255.255.0 “push “route 66.249.93.0 255.255.255.0 “push “route 66.249.69.0 255.255.255.0 “push “route 74.125.25.0 255.255.255.0 “push “route 74.125.228.0 255.255.255.0 “push “route 64.233.165.0 255.255.255.0 “push “route 216.239.36.0 255.255.255.0 “push “route 173.194.91.0 255.255.255.0 “push “route 74.125.21.0 255.255.255.0 “push “route 74.125.73.0 255.255.255.0 “push “route 216.239.32.0 255.255.255.0 “push “route 108.170.192.0 255.255.192.0 “push “route 199.223.232.0 255.255.248.0 “push “route 74.125.225.0 255.255.255.0 “push “route 199.192.112.0 255.255.252.0 “push “route 173.194.77.0 255.255.255.0 “push “route 66.249.73.0 255.255.255.0 “push “route 66.249.81.0 255.255.255.0 “push “route 64.233.167.0 255.255.255.0 “push “route 8.8.4.0 255.255.255.0 “push “route 74.125.72.0 255.255.255.0 “push “route 74.125.135.0 255.255.255.0 “push “route 74.125.45.0 255.255.255.0 “push “route 216.239.33.0 255.255.255.0 “push “route 107.167.160.0 255.255.224.0 “push “route 173.194.38.0 255.255.255.0 “push “route 1.1.1.0 255.255.255.0 “push “route 74.125.195.0 255.255.255.0 “push “route 74.125.204.0 255.255.255.0 “push “route 173.194.126.0 255.255.255.0 “push “route 74.125.23.0 255.255.255.0 “push “route 66.249.66.0 255.255.255.0 “push “route 66.249.76.0 255.255.255.0 “push “route 173.194.71.0 255.255.255.0 “push “route 74.125.181.0 255.255.255.0 “# add opendns route rulepush “route 208.67.0.0 255.255.0.0 " 客户端配置：```bash clientdev tunproto udpremote xx.xx.xx.xx 60xxxresolv-retry infinitenobindpersist-keypersist-tunca openvzca.crtcert openvz.crtkey openvz.keyns-cert-type servercomp-lzoverb 3mute 20keepalive 20 60;redirect-gatewaymax-routes 10000

一个因VLAN tag和MTU引发的线上故障

前段时间帮其他部门的同事排查了一个故障和MTU有点关系。同事找到我说线上的服务器访问有点问题，并直接给出了关键点:使用ping的时候 -s 1468能通，但是-s 1469就不通了。简单看了一下是个虚拟机，再综合宿主机的情况看了下。就判断出是在虚拟机上的网卡加vlan tag引发的问题。因为我们习惯性的都是在宿主机上打vlan tag，单独建个网桥，并把虚拟机的网卡桥接到这个网桥上。让同事把对应的VM业务停掉后，按照我自己的方式重新修改了宿主机和VM的配置文件，启动后一切正常。从原理上分析，vlan tag占用4个字节，以太网的MTU最大可以设置1500.当在VM内的包如果直接发了1469,那么就会造成1469+4(vlan tag)+20(ip头)+8(icmp头)>1500，就会被分片，可能在分配重组的时候丢掉了vlan 信息，造成通信异常。解决方案无非2种。1. 完全重新修改宿主机的配置，重新生成VM。vlan tag在宿主机上打.2. 修改VM网卡的MTU设置，设置为1496.这样通过MSS协商，TCP的通信会协商MSS为1496-40=1456.可以保证TCP通信正常。

使用route-map修改收到的路由

在quagga内，可以配置zebra使得对外部OSPF/BGP学习到的路由进行一些处理。比如学到了1段到192.168.0.0/16的路由，如果想自己单独指定到这段地址时使用特定IP做源IP，那么可以直接在配置 ip prefix-list INNET1 seq 5 permit 192.168.0.0/16 le 32 route-map Server2INNET1 permit 10 match ip address prefix-list INNET1 set src 10.10.7.6 这样当收到102.168.0.0/16这段路由时，实际注入kernel的时候会加上src,变成类似 192.168.0.0/16 dev eth0 proto kernel scope link src 10.10.7.6 metric 11 此时如果服务器去访问192.168.0.0/16的地址时，便自动使用10.10.7.6这个IP做源IP。也可以直接默认的源IP改掉，这样默认都使用这个IP做源IP出去。 ip prefix-list DEFAULT seq 5 permit 0.0.0.0/0 ! route-map Server2default permit 10 match ip address prefix-list DEFAULT set src 10.10.7.6 也可以在zebra内添加静态的路由，修改到特定目标网段的时使用的源IP ip route 10.0.0.0/8 10.10.6.5 ip route 10.0.0.0/8 10.10.7.5 ip route 172.16.0.0/12 10.10.6.5 ip route 172.16.0.0/12 10.10.7.5 ip prefix-list static seq 10 permit 172.16.0.0/12 ip prefix-list default seq 5 permit 0.0.0.0/0 ...

单机跑20G带宽

大家知道2个千兆网卡做bond，mode4的时候因为是两个网卡都会有流量，出口的带宽总和就是网卡数量X单网卡带宽。一般双网卡的机器我们的上联带宽就是2Gbps了。如果2个网卡都是万兆的，当处于万兆pod内，就可以通过做bond来使得单机具备20G的出口带宽。但如果服务器是和上联的交换机跑ospf的，默认的情况下只能学到1条默认路由，出口的带宽就只有10G了。我们可以通过打开quagga的multipath，使得能同时学习到2条默认路由。首先需要确认我们的内核是支持IP_ROUTE_MULTIPATH 的 #grep CONFIG_IP_ROUTE_MULTIPATH /boot/config-uname -r CONFIG_IP_ROUTE_MULTIPATH=y 其次就是编译quagga的时候打开multipath了。 ./configure –disable-ipv6 –enable-multipath=2 - 启动后可以看到能学到2条metric值一样的路由 default proto zebra metric 11 nexthop via 192.10.193.5 dev eth4 weight 1 nexthop via 192.10.194.5 dev eth5 weight 1 实际测试单机可以跑到18G左右的流量

小型高可用NAT网关设计

小企业内，很多都是直接拿1台Linux服务器用iptables做NAT，给内部服务器提供上网的需求。但是这样的NAT网关实际很容易成为单点。所以在设计的时候最好还是需要做一下HA。简单的考虑小规模场景，可以2个服务器起一个keepalived跑VRRP，把网关的IP和SNAT的IP都做浮动IP。这样单机挂掉的时候能自动进行切换。如果公网地址足够，最好是需用一个段做SNAT地址池。比如64个地址的话就ifcfg-eth0:1~ifcfg-eth0:64，每个配置文件把公网地址配置上去 DEVICE="eth0.X:Y" BOOTPROTO="static" ONBOOT="yes" TYPE="ethernet" IPADDR=aa.aa.aa.Y NETMASK=255.255.255.0 VLAN=yes PEERDNS=no /etc/sysctl.conf记得配置net.ipv4.ip_forward = 1。 iptables规则如下 *filter :INPUT ACCEPT [3:309] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [518:57577] -A INPUT -m state --state RELATED,ESTABLISHED,UNTRACKED -j ACCEPT -A INPUT -d a.a.a.0/255.255.255.0 -i eth0.X -p tcp -j DROP -A INPUT -d a.a.a.0/255.255.255.0 -i eth0.X -p udp -j DROP COMMIT Completed on Tue Apr 15 18:21:48 2014 *nat :PREROUTING ACCEPT [6778:567991] :POSTROUTING ACCEPT [41:3373] :OUTPUT ACCEPT [41:3373] -A POSTROUTING -s 10.10.0.0/16 -o eth0.X-j SNAT --to- source aa.aa.aa.1-aa.aa.aa.64 COMMIT ...

开放的BGP路由器

有时需要能查一些BGP的AS PATH信息，有一个“Route Views Project”项目。可以直接登录到一些zebra上直接查一下。bgp的命令可以参考quagga的手册，简单的比如：1. 查看某个网段的bgp信息 “Route Views Project” show ip bgp 8.8.8.8/24 route-views.isc.routeviews.org> show ip bgp 8.8.8.8/24 BGP routing table entry for 8.8.8.0/24 Paths: (11 available, best #11, table Default-IP-Routing-Table) Not advertised to any peer 4436 209 15169 198.32.176.13 from 198.32.176.13 (69.22.143.244) Origin IGP, metric 13, localpref 100, valid, external Community: 209:209 209:888 209:40822 4436:999 4436:31413 Last update: Mon Apr 7 12:08:01 2014 14361 15169 198.32.176.10 from 198.32.176.10 (66.36.224.11) Origin IGP, localpref 100, valid, external Last update: Thu Apr 3 19:28:25 2014 ...

根据AS号查询IP段

he.net上可以直接看到任何AS所归属的IP段。可以写个脚本查询一下，默认是把curl屏蔽了的可以改一下UA。 #!/usr/bin/python import urllib2 import sys import os import re import string class AS_TO_ACL: def __init__(self,asnum): self.asnum=asnum; self.url="http://bgp.he.net/AS%s#_prefixes"%(self.asnum) self.cidr=set(); def http_client(self,url): request=urllib2.Request(url,headers={'User-agent':"Chrome 27.0"}) try: response=urllib2.urlopen(request,timeout=5) info=response.info() data=response.read() except urllib2.HTTPError,error: print “%s error:%s” %(url,error.reason) return None except urllib2.URLError,error: print error.reason return None else: outdata=data return outdata def get_acl(self): htmldata=self.http_client(self.url) ip_reg=re.compile("/net/(\d+\.\d+.\d+\.\d+/\d+)") htmls=htmldata.split(); for line in htmls: match=ip_reg.search(line) if match: ips=match.group(1) self.cidr.add(string.strip(ips)) for ip in self.cidr: print “%s;\n”%(ip), if len(sys.argv)<2: print “error!” print “as_to_acl ASN” sys.exit(1) query=AS_TO_ACL(sys.argv[1]) query.get_acl() ...

tc做网络延迟、丢包模拟

在生产环境上经常需要模拟一下应用间异地调用的场景，或者模拟一下协议栈优化在丢包的情况下是否有效果。可以使用netem来做。具体的直接看看的介绍就行。 Examples Emulating wide area network delays This is the simplest example, it just adds a fixed amount of delay to all packets going out of the local Ethernet. Real wide area networks show variability so it is possible to add random variation. tc qdisc change dev eth0 root netem delay 100ms 10msThis causes the added delay to be 100ms ± 10ms. Network delay variation isn’t purely random, so to emulate that there is a correlation value as well. tc qdisc change dev eth0 root netem delay 100ms 10ms 25%This causes the added delay to be 100ms ± 10ms with the next random element depending 25% on the last one. This isn’t true statistical correlation, but an approximation. Delay distribution ...