AWS にて、 dockerの立ち上げがうまく行かなかった

AWSにて、「データ分析基盤構築入門」を見て、立ち上げたけどエラーが出た。。

https://www.amazon.co.jp/dp/B075RTZ141

$ git clone https://github.com/efkbook/blog-sample

$ cd blog-sample/

$ docker-compose up -d
Building go
Step 1/5 : FROM golang:1.8
1.8: Pulling from library/golang
4176fe04cefe: Pull complete
851356ecf618: Pull complete
6115379c7b49: Pull complete
69914558965c: Pull complete
b108f9aa98db: Pull complete
df7abcd2981e: Pull complete
3e60cb3f592b: Pull complete
Digest: sha256:f0b5dab7581eddb49dabd1d1b9aa505ca3edcdf79a66395b5bfa4f3c036b49ef
Status: Downloaded newer image for golang:1.8
 ---> 0d283eb41a92
Step 2/5 : RUN mkdir -p /go/src/github.com/efkbook/blog-sample
 ---> Running in 59dba2dba710
 ---> 2fcb72cc73c5
Removing intermediate container 59dba2dba710
Step 3/5 : WORKDIR /go/src/github.com/efkbook/blog-sample
 ---> 9719d83418fe
Removing intermediate container cdab235d0dd4
Step 4/5 : CMD make app/run
 ---> Running in ae90cd6f44cd
 ---> e6a2ebc9eaa0
Removing intermediate container ae90cd6f44cd
Step 5/5 : EXPOSE 8080
 ---> Running in a3e82ed29f86
 ---> c50bc29f92a4
Removing intermediate container a3e82ed29f86
Successfully built c50bc29f92a4
Successfully tagged blogsample_go:latest
WARNING: Image for service go was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Building fluentd
Step 1/7 : FROM debian:jessie
jessie: Pulling from library/debian
f2b6b4884fc8: Pull complete
Digest: sha256:a69efd2cf1f83d7fe6a374aaf27974ed031d72b2886acb4b730c6a2017db0c3d
Status: Downloaded newer image for debian:jessie
 ---> 5dd74d62fab8
Step 2/7 : ENV DEBIAN_FRONTEND noninteractive
 ---> Running in 6bd988af2ec6
 ---> 3cb72631b1ab
Removing intermediate container 6bd988af2ec6
Step 3/7 : RUN apt-get -qq update && apt-get install --no-install-recommends -y curl ca-certificates sudo build-essential libcurl4-gnutls-dev
 ---> Running in f59911d3c822
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  binutils bzip2 cpp cpp-4.9 dpkg-dev g++ g++-4.9 gcc gcc-4.9 libasan1
  libatomic1 libc-dev-bin libc6-dev libcilkrts5 libcloog-isl4 libcurl3
  libcurl3-gnutls libdpkg-perl libffi6 libgcc-4.9-dev libgdbm3 libgmp10
  libgnutls-deb0-28 libgomp1 libgssapi-krb5-2 libhogweed2 libidn11 libisl10
  libitm1 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2
  liblsan0 libmpc3 libmpfr4 libnettle4 libp11-kit0 libquadmath0 librtmp1
  libsasl2-2 libsasl2-modules-db libssh2-1 libssl1.0.0 libstdc++-4.9-dev
  libtasn1-6 libtimedate-perl libtsan0 libubsan0 linux-libc-dev make openssl
  patch perl perl-modules xz-utils
Suggested packages:
  binutils-doc bzip2-doc cpp-doc gcc-4.9-locales debian-keyring g++-multilib
  g++-4.9-multilib gcc-4.9-doc libstdc++6-4.9-dbg gcc-multilib manpages-dev
  autoconf automake libtool flex bison gdb gcc-doc gcc-4.9-multilib
  libgcc1-dbg libgomp1-dbg libitm1-dbg libatomic1-dbg libasan1-dbg
  liblsan0-dbg libtsan0-dbg libubsan0-dbg libcilkrts5-dbg libquadmath0-dbg
  glibc-doc libcurl4-doc libcurl3-dbg libgnutls28-dev libidn11-dev libkrb5-dev
  libldap2-dev librtmp-dev libssh2-1-dev pkg-config zlib1g-dev gnutls-bin
  krb5-doc krb5-user libstdc++-4.9-doc make-doc ed diffutils-doc perl-doc
  libterm-readline-gnu-perl libterm-readline-perl-perl libb-lint-perl
  libcpanplus-dist-build-perl libcpanplus-perl libfile-checktree-perl
  liblog-message-simple-perl liblog-message-perl libobject-accessor-perl
Recommended packages:
  fakeroot libalgorithm-merge-perl libfile-fcntllock-perl krb5-locales
  libsasl2-modules rename libarchive-extract-perl libmodule-pluggable-perl
  libpod-latex-perl libterm-ui-perl libtext-soundex-perl libcgi-pm-perl
  libmodule-build-perl libpackage-constants-perl
The following NEW packages will be installed:
  binutils build-essential bzip2 ca-certificates cpp cpp-4.9 curl dpkg-dev g++
  g++-4.9 gcc gcc-4.9 libasan1 libatomic1 libc-dev-bin libc6-dev libcilkrts5
  libcloog-isl4 libcurl3 libcurl3-gnutls libcurl4-gnutls-dev libdpkg-perl
  libffi6 libgcc-4.9-dev libgdbm3 libgmp10 libgnutls-deb0-28 libgomp1
  libgssapi-krb5-2 libhogweed2 libidn11 libisl10 libitm1 libk5crypto3
  libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2 liblsan0 libmpc3
  libmpfr4 libnettle4 libp11-kit0 libquadmath0 librtmp1 libsasl2-2
  libsasl2-modules-db libssh2-1 libssl1.0.0 libstdc++-4.9-dev libtasn1-6
  libtimedate-perl libtsan0 libubsan0 linux-libc-dev make openssl patch perl
  perl-modules sudo xz-utils
0 upgraded, 62 newly installed, 0 to remove and 1 not upgraded.
Need to get 54.6 MB of archives.
After this operation, 186 MB of additional disk space will be used.
Get:1 http://security.debian.org/ jessie/updates/main libssl1.0.0 amd64 1.0.1t-1+deb8u8 [1044 kB]
Get:2 http://security.debian.org/ jessie/updates/main libasan1 amd64 4.9.2-10+deb8u1 [195 kB]
Get:3 http://security.debian.org/ jessie/updates/main libatomic1 amd64 4.9.2-10+deb8u1 [9014 B]
Get:4 http://security.debian.org/ jessie/updates/main libcilkrts5 amd64 4.9.2-10+deb8u1 [40.1 kB]
Get:5 http://security.debian.org/ jessie/updates/main libcurl3 amd64 7.38.0-4+deb8u10 [260 kB]
Get:6 http://security.debian.org/ jessie/updates/main libcurl3-gnutls amd64 7.38.0-4+deb8u10 [252 kB]
Get:7 http://security.debian.org/ jessie/updates/main libgomp1 amd64 4.9.2-10+deb8u1 [37.8 kB]
Get:8 http://security.debian.org/ jessie/updates/main libitm1 amd64 4.9.2-10+deb8u1 [29.3 kB]
Get:9 http://security.debian.org/ jessie/updates/main liblsan0 amd64 4.9.2-10+deb8u1 [92.7 kB]
Get:10 http://security.debian.org/ jessie/updates/main libquadmath0 amd64 4.9.2-10+deb8u1 [129 kB]
Get:11 http://security.debian.org/ jessie/updates/main libtsan0 amd64 4.9.2-10+deb8u1 [212 kB]
Get:12 http://security.debian.org/ jessie/updates/main libubsan0 amd64 4.9.2-10+deb8u1 [82.4 kB]
Get:13 http://security.debian.org/ jessie/updates/main cpp-4.9 amd64 4.9.2-10+deb8u1 [5002 kB]
Get:14 http://security.debian.org/ jessie/updates/main libgcc-4.9-dev amd64 4.9.2-10+deb8u1 [2066 kB]
Get:15 http://security.debian.org/ jessie/updates/main linux-libc-dev amd64 3.16.51-3+deb8u1 [1083 kB]
Get:16 http://deb.debian.org/debian/ jessie/main libgdbm3 amd64 1.8.3-13.1 [30.0 kB]
Get:17 http://security.debian.org/ jessie/updates/main gcc-4.9 amd64 4.9.2-10+deb8u1 [5184 kB]
Get:18 http://security.debian.org/ jessie/updates/main libstdc++-4.9-dev amd64 4.9.2-10+deb8u1 [1120 kB]
Get:19 http://security.debian.org/ jessie/updates/main g++-4.9 amd64 4.9.2-10+deb8u1 [17.3 MB]
Get:20 http://deb.debian.org/debian/ jessie/main libgmp10 amd64 2:6.0.0+dfsg-6 [253 kB]
Get:21 http://security.debian.org/ jessie/updates/main openssl amd64 1.0.1t-1+deb8u8 [664 kB]
Get:22 http://security.debian.org/ jessie/updates/main curl amd64 7.38.0-4+deb8u10 [201 kB]
Get:23 http://security.debian.org/ jessie/updates/main libcurl4-gnutls-dev amd64 7.38.0-4+deb8u10 [329 kB]
Get:24 http://deb.debian.org/debian/ jessie/main libnettle4 amd64 2.7.1-5+deb8u2 [176 kB]
Get:25 http://deb.debian.org/debian/ jessie/main libhogweed2 amd64 2.7.1-5+deb8u2 [125 kB]
Get:26 http://deb.debian.org/debian/ jessie/main libffi6 amd64 3.1-2+deb8u1 [20.2 kB]
Get:27 http://deb.debian.org/debian/ jessie/main libp11-kit0 amd64 0.20.7-1 [81.2 kB]
Get:28 http://deb.debian.org/debian/ jessie/main libtasn1-6 amd64 4.2-3+deb8u3 [49.2 kB]
Get:29 http://deb.debian.org/debian/ jessie/main libgnutls-deb0-28 amd64 3.3.8-6+deb8u7 [696 kB]
Get:30 http://deb.debian.org/debian/ jessie/main libkeyutils1 amd64 1.5.9-5+b1 [12.0 kB]
Get:31 http://deb.debian.org/debian/ jessie/main libkrb5support0 amd64 1.12.1+dfsg-19+deb8u4 [59.4 kB]
Get:32 http://deb.debian.org/debian/ jessie/main libk5crypto3 amd64 1.12.1+dfsg-19+deb8u4 [116 kB]
Get:33 http://deb.debian.org/debian/ jessie/main libkrb5-3 amd64 1.12.1+dfsg-19+deb8u4 [303 kB]
Get:34 http://deb.debian.org/debian/ jessie/main libgssapi-krb5-2 amd64 1.12.1+dfsg-19+deb8u4 [152 kB]
Get:35 http://deb.debian.org/debian/ jessie/main libidn11 amd64 1.29-1+deb8u2 [136 kB]
Get:36 http://deb.debian.org/debian/ jessie/main libsasl2-modules-db amd64 2.1.26.dfsg1-13+deb8u1 [67.1 kB]
Get:37 http://deb.debian.org/debian/ jessie/main libsasl2-2 amd64 2.1.26.dfsg1-13+deb8u1 [105 kB]
Get:38 http://deb.debian.org/debian/ jessie/main libldap-2.4-2 amd64 2.4.40+dfsg-1+deb8u3 [218 kB]
Get:39 http://deb.debian.org/debian/ jessie/main perl-modules all 5.20.2-3+deb8u9 [2553 kB]
Get:40 http://deb.debian.org/debian/ jessie/main perl amd64 5.20.2-3+deb8u9 [2641 kB]
Get:41 http://deb.debian.org/debian/ jessie/main libisl10 amd64 0.12.2-2 [440 kB]
Get:42 http://deb.debian.org/debian/ jessie/main libcloog-isl4 amd64 0.18.2-1+b2 [61.8 kB]
Get:43 http://deb.debian.org/debian/ jessie/main librtmp1 amd64 2.4+20150115.gita107cef-1+deb8u1 [60.0 kB]
Get:44 http://deb.debian.org/debian/ jessie/main libssh2-1 amd64 1.4.3-4.1+deb8u1 [125 kB]
Get:45 http://deb.debian.org/debian/ jessie/main libmpfr4 amd64 3.1.2-2 [527 kB]
Get:46 http://deb.debian.org/debian/ jessie/main libmpc3 amd64 1.0.2-1 [39.3 kB]
Get:47 http://deb.debian.org/debian/ jessie/main bzip2 amd64 1.0.6-7+b3 [46.9 kB]
Get:48 http://deb.debian.org/debian/ jessie/main patch amd64 2.7.5-1 [109 kB]
Get:49 http://deb.debian.org/debian/ jessie/main xz-utils amd64 5.1.1alpha+20120614-2+b3 [221 kB]
Get:50 http://deb.debian.org/debian/ jessie/main binutils amd64 2.25-5+deb8u1 [3496 kB]
Get:51 http://deb.debian.org/debian/ jessie/main libc-dev-bin amd64 2.19-18+deb8u10 [238 kB]
Get:52 http://deb.debian.org/debian/ jessie/main libc6-dev amd64 2.19-18+deb8u10 [2003 kB]
Get:53 http://deb.debian.org/debian/ jessie/main cpp amd64 4:4.9.2-2 [17.3 kB]
Get:54 http://deb.debian.org/debian/ jessie/main gcc amd64 4:4.9.2-2 [5136 B]
Get:55 http://deb.debian.org/debian/ jessie/main g++ amd64 4:4.9.2-2 [1530 B]
Get:56 http://deb.debian.org/debian/ jessie/main make amd64 4.0-8.1 [349 kB]
Get:57 http://deb.debian.org/debian/ jessie/main libtimedate-perl all 2.3000-2 [42.2 kB]
Get:58 http://deb.debian.org/debian/ jessie/main libdpkg-perl all 1.17.27 [1075 kB]
Get:59 http://deb.debian.org/debian/ jessie/main dpkg-dev all 1.17.27 [1548 kB]
Get:60 http://deb.debian.org/debian/ jessie/main build-essential amd64 11.7 [7114 B]
Get:61 http://deb.debian.org/debian/ jessie/main ca-certificates all 20141019+deb8u3 [207 kB]
Get:62 http://deb.debian.org/debian/ jessie/main sudo amd64 1.8.10p3-1+deb8u5 [854 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 54.6 MB in 12s (4386 kB/s)
Selecting previously unselected package libgdbm3:amd64.
(Reading database ... 7561 files and directories currently installed.)
Preparing to unpack .../libgdbm3_1.8.3-13.1_amd64.deb ...
Unpacking libgdbm3:amd64 (1.8.3-13.1) ...
Selecting previously unselected package libssl1.0.0:amd64.
Preparing to unpack .../libssl1.0.0_1.0.1t-1+deb8u8_amd64.deb ...
Unpacking libssl1.0.0:amd64 (1.0.1t-1+deb8u8) ...
Selecting previously unselected package libgmp10:amd64.
Preparing to unpack .../libgmp10_2%3a6.0.0+dfsg-6_amd64.deb ...
Unpacking libgmp10:amd64 (2:6.0.0+dfsg-6) ...
Selecting previously unselected package libnettle4:amd64.
Preparing to unpack .../libnettle4_2.7.1-5+deb8u2_amd64.deb ...
Unpacking libnettle4:amd64 (2.7.1-5+deb8u2) ...
Selecting previously unselected package libhogweed2:amd64.
Preparing to unpack .../libhogweed2_2.7.1-5+deb8u2_amd64.deb ...
Unpacking libhogweed2:amd64 (2.7.1-5+deb8u2) ...
Selecting previously unselected package libffi6:amd64.
Preparing to unpack .../libffi6_3.1-2+deb8u1_amd64.deb ...
Unpacking libffi6:amd64 (3.1-2+deb8u1) ...
Selecting previously unselected package libp11-kit0:amd64.
Preparing to unpack .../libp11-kit0_0.20.7-1_amd64.deb ...
Unpacking libp11-kit0:amd64 (0.20.7-1) ...
Selecting previously unselected package libtasn1-6:amd64.
Preparing to unpack .../libtasn1-6_4.2-3+deb8u3_amd64.deb ...
Unpacking libtasn1-6:amd64 (4.2-3+deb8u3) ...
Selecting previously unselected package libgnutls-deb0-28:amd64.
Preparing to unpack .../libgnutls-deb0-28_3.3.8-6+deb8u7_amd64.deb ...
Unpacking libgnutls-deb0-28:amd64 (3.3.8-6+deb8u7) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../libkeyutils1_1.5.9-5+b1_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.5.9-5+b1) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../libkrb5support0_1.12.1+dfsg-19+deb8u4_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.12.1+dfsg-19+deb8u4) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../libk5crypto3_1.12.1+dfsg-19+deb8u4_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.12.1+dfsg-19+deb8u4) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../libkrb5-3_1.12.1+dfsg-19+deb8u4_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.12.1+dfsg-19+deb8u4) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../libgssapi-krb5-2_1.12.1+dfsg-19+deb8u4_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.12.1+dfsg-19+deb8u4) ...
Selecting previously unselected package libidn11:amd64.
Preparing to unpack .../libidn11_1.29-1+deb8u2_amd64.deb ...
Unpacking libidn11:amd64 (1.29-1+deb8u2) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../libsasl2-modules-db_2.1.26.dfsg1-13+deb8u1_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.26.dfsg1-13+deb8u1) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../libsasl2-2_2.1.26.dfsg1-13+deb8u1_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.26.dfsg1-13+deb8u1) ...
Selecting previously unselected package libldap-2.4-2:amd64.
Preparing to unpack .../libldap-2.4-2_2.4.40+dfsg-1+deb8u3_amd64.deb ...
Unpacking libldap-2.4-2:amd64 (2.4.40+dfsg-1+deb8u3) ...
Selecting previously unselected package perl-modules.
Preparing to unpack .../perl-modules_5.20.2-3+deb8u9_all.deb ...
Unpacking perl-modules (5.20.2-3+deb8u9) ...
Selecting previously unselected package perl.
Preparing to unpack .../perl_5.20.2-3+deb8u9_amd64.deb ...
Unpacking perl (5.20.2-3+deb8u9) ...
Selecting previously unselected package libasan1:amd64.
Preparing to unpack .../libasan1_4.9.2-10+deb8u1_amd64.deb ...
Unpacking libasan1:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package libatomic1:amd64.
Preparing to unpack .../libatomic1_4.9.2-10+deb8u1_amd64.deb ...
Unpacking libatomic1:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package libcilkrts5:amd64.
Preparing to unpack .../libcilkrts5_4.9.2-10+deb8u1_amd64.deb ...
Unpacking libcilkrts5:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package libisl10:amd64.
Preparing to unpack .../libisl10_0.12.2-2_amd64.deb ...
Unpacking libisl10:amd64 (0.12.2-2) ...
Selecting previously unselected package libcloog-isl4:amd64.
Preparing to unpack .../libcloog-isl4_0.18.2-1+b2_amd64.deb ...
Unpacking libcloog-isl4:amd64 (0.18.2-1+b2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../librtmp1_2.4+20150115.gita107cef-1+deb8u1_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20150115.gita107cef-1+deb8u1) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../libssh2-1_1.4.3-4.1+deb8u1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.4.3-4.1+deb8u1) ...
Selecting previously unselected package libcurl3:amd64.
Preparing to unpack .../libcurl3_7.38.0-4+deb8u10_amd64.deb ...
Unpacking libcurl3:amd64 (7.38.0-4+deb8u10) ...
Selecting previously unselected package libcurl3-gnutls:amd64.
Preparing to unpack .../libcurl3-gnutls_7.38.0-4+deb8u10_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.38.0-4+deb8u10) ...
Selecting previously unselected package libgomp1:amd64.
Preparing to unpack .../libgomp1_4.9.2-10+deb8u1_amd64.deb ...
Unpacking libgomp1:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package libitm1:amd64.
Preparing to unpack .../libitm1_4.9.2-10+deb8u1_amd64.deb ...
Unpacking libitm1:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package liblsan0:amd64.
Preparing to unpack .../liblsan0_4.9.2-10+deb8u1_amd64.deb ...
Unpacking liblsan0:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package libmpfr4:amd64.
Preparing to unpack .../libmpfr4_3.1.2-2_amd64.deb ...
Unpacking libmpfr4:amd64 (3.1.2-2) ...
Selecting previously unselected package libquadmath0:amd64.
Preparing to unpack .../libquadmath0_4.9.2-10+deb8u1_amd64.deb ...
Unpacking libquadmath0:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package libtsan0:amd64.
Preparing to unpack .../libtsan0_4.9.2-10+deb8u1_amd64.deb ...
Unpacking libtsan0:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package libubsan0:amd64.
Preparing to unpack .../libubsan0_4.9.2-10+deb8u1_amd64.deb ...
Unpacking libubsan0:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package libmpc3:amd64.
Preparing to unpack .../libmpc3_1.0.2-1_amd64.deb ...
Unpacking libmpc3:amd64 (1.0.2-1) ...
Selecting previously unselected package bzip2.
Preparing to unpack .../bzip2_1.0.6-7+b3_amd64.deb ...
Unpacking bzip2 (1.0.6-7+b3) ...
Selecting previously unselected package patch.
Preparing to unpack .../patch_2.7.5-1_amd64.deb ...
Unpacking patch (2.7.5-1) ...
Selecting previously unselected package xz-utils.
Preparing to unpack .../xz-utils_5.1.1alpha+20120614-2+b3_amd64.deb ...
Unpacking xz-utils (5.1.1alpha+20120614-2+b3) ...
Selecting previously unselected package binutils.
Preparing to unpack .../binutils_2.25-5+deb8u1_amd64.deb ...
Unpacking binutils (2.25-5+deb8u1) ...
Selecting previously unselected package libc-dev-bin.
Preparing to unpack .../libc-dev-bin_2.19-18+deb8u10_amd64.deb ...
Unpacking libc-dev-bin (2.19-18+deb8u10) ...
Selecting previously unselected package linux-libc-dev:amd64.
Preparing to unpack .../linux-libc-dev_3.16.51-3+deb8u1_amd64.deb ...
Unpacking linux-libc-dev:amd64 (3.16.51-3+deb8u1) ...
Selecting previously unselected package libc6-dev:amd64.
Preparing to unpack .../libc6-dev_2.19-18+deb8u10_amd64.deb ...
Unpacking libc6-dev:amd64 (2.19-18+deb8u10) ...
Selecting previously unselected package cpp-4.9.
Preparing to unpack .../cpp-4.9_4.9.2-10+deb8u1_amd64.deb ...
Unpacking cpp-4.9 (4.9.2-10+deb8u1) ...
Selecting previously unselected package cpp.
Preparing to unpack .../cpp_4%3a4.9.2-2_amd64.deb ...
Unpacking cpp (4:4.9.2-2) ...
Selecting previously unselected package libgcc-4.9-dev:amd64.
Preparing to unpack .../libgcc-4.9-dev_4.9.2-10+deb8u1_amd64.deb ...
Unpacking libgcc-4.9-dev:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package gcc-4.9.
Preparing to unpack .../gcc-4.9_4.9.2-10+deb8u1_amd64.deb ...
Unpacking gcc-4.9 (4.9.2-10+deb8u1) ...
Selecting previously unselected package gcc.
Preparing to unpack .../gcc_4%3a4.9.2-2_amd64.deb ...
Unpacking gcc (4:4.9.2-2) ...
Selecting previously unselected package libstdc++-4.9-dev:amd64.
Preparing to unpack .../libstdc++-4.9-dev_4.9.2-10+deb8u1_amd64.deb ...
Unpacking libstdc++-4.9-dev:amd64 (4.9.2-10+deb8u1) ...
Selecting previously unselected package g++-4.9.
Preparing to unpack .../g++-4.9_4.9.2-10+deb8u1_amd64.deb ...
Unpacking g++-4.9 (4.9.2-10+deb8u1) ...
Selecting previously unselected package g++.
Preparing to unpack .../g++_4%3a4.9.2-2_amd64.deb ...
Unpacking g++ (4:4.9.2-2) ...
Selecting previously unselected package make.
Preparing to unpack .../make_4.0-8.1_amd64.deb ...
Unpacking make (4.0-8.1) ...
Selecting previously unselected package libtimedate-perl.
Preparing to unpack .../libtimedate-perl_2.3000-2_all.deb ...
Unpacking libtimedate-perl (2.3000-2) ...
Selecting previously unselected package libdpkg-perl.
Preparing to unpack .../libdpkg-perl_1.17.27_all.deb ...
Unpacking libdpkg-perl (1.17.27) ...
Selecting previously unselected package dpkg-dev.
Preparing to unpack .../dpkg-dev_1.17.27_all.deb ...
Unpacking dpkg-dev (1.17.27) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../build-essential_11.7_amd64.deb ...
Unpacking build-essential (11.7) ...
Selecting previously unselected package openssl.
Preparing to unpack .../openssl_1.0.1t-1+deb8u8_amd64.deb ...
Unpacking openssl (1.0.1t-1+deb8u8) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../ca-certificates_20141019+deb8u3_all.deb ...
Unpacking ca-certificates (20141019+deb8u3) ...
Selecting previously unselected package curl.
Preparing to unpack .../curl_7.38.0-4+deb8u10_amd64.deb ...
Unpacking curl (7.38.0-4+deb8u10) ...
Selecting previously unselected package libcurl4-gnutls-dev:amd64.
Preparing to unpack .../libcurl4-gnutls-dev_7.38.0-4+deb8u10_amd64.deb ...
Unpacking libcurl4-gnutls-dev:amd64 (7.38.0-4+deb8u10) ...
Selecting previously unselected package sudo.
Preparing to unpack .../sudo_1.8.10p3-1+deb8u5_amd64.deb ...
Unpacking sudo (1.8.10p3-1+deb8u5) ...
Processing triggers for systemd (215-17+deb8u7) ...
Setting up libgdbm3:amd64 (1.8.3-13.1) ...
Setting up libssl1.0.0:amd64 (1.0.1t-1+deb8u8) ...
Setting up libgmp10:amd64 (2:6.0.0+dfsg-6) ...
Setting up libnettle4:amd64 (2.7.1-5+deb8u2) ...
Setting up libhogweed2:amd64 (2.7.1-5+deb8u2) ...
Setting up libffi6:amd64 (3.1-2+deb8u1) ...
Setting up libp11-kit0:amd64 (0.20.7-1) ...
Setting up libtasn1-6:amd64 (4.2-3+deb8u3) ...
Setting up libgnutls-deb0-28:amd64 (3.3.8-6+deb8u7) ...
Setting up libkeyutils1:amd64 (1.5.9-5+b1) ...
Setting up libkrb5support0:amd64 (1.12.1+dfsg-19+deb8u4) ...
Setting up libk5crypto3:amd64 (1.12.1+dfsg-19+deb8u4) ...
Setting up libkrb5-3:amd64 (1.12.1+dfsg-19+deb8u4) ...
Setting up libgssapi-krb5-2:amd64 (1.12.1+dfsg-19+deb8u4) ...
Setting up libidn11:amd64 (1.29-1+deb8u2) ...
Setting up libsasl2-modules-db:amd64 (2.1.26.dfsg1-13+deb8u1) ...
Setting up libsasl2-2:amd64 (2.1.26.dfsg1-13+deb8u1) ...
Setting up libldap-2.4-2:amd64 (2.4.40+dfsg-1+deb8u3) ...
Setting up perl-modules (5.20.2-3+deb8u9) ...
Setting up perl (5.20.2-3+deb8u9) ...
update-alternatives: using /usr/bin/prename to provide /usr/bin/rename (rename) in auto mode
Setting up libasan1:amd64 (4.9.2-10+deb8u1) ...
Setting up libatomic1:amd64 (4.9.2-10+deb8u1) ...
Setting up libcilkrts5:amd64 (4.9.2-10+deb8u1) ...
Setting up libisl10:amd64 (0.12.2-2) ...
Setting up libcloog-isl4:amd64 (0.18.2-1+b2) ...
Setting up librtmp1:amd64 (2.4+20150115.gita107cef-1+deb8u1) ...
Setting up libssh2-1:amd64 (1.4.3-4.1+deb8u1) ...
Setting up libcurl3:amd64 (7.38.0-4+deb8u10) ...
Setting up libcurl3-gnutls:amd64 (7.38.0-4+deb8u10) ...
Setting up libgomp1:amd64 (4.9.2-10+deb8u1) ...
Setting up libitm1:amd64 (4.9.2-10+deb8u1) ...
Setting up liblsan0:amd64 (4.9.2-10+deb8u1) ...
Setting up libmpfr4:amd64 (3.1.2-2) ...
Setting up libquadmath0:amd64 (4.9.2-10+deb8u1) ...
Setting up libtsan0:amd64 (4.9.2-10+deb8u1) ...
Setting up libubsan0:amd64 (4.9.2-10+deb8u1) ...
Setting up libmpc3:amd64 (1.0.2-1) ...
Setting up bzip2 (1.0.6-7+b3) ...
Setting up patch (2.7.5-1) ...
Setting up xz-utils (5.1.1alpha+20120614-2+b3) ...
update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
Setting up binutils (2.25-5+deb8u1) ...
Setting up libc-dev-bin (2.19-18+deb8u10) ...
Setting up linux-libc-dev:amd64 (3.16.51-3+deb8u1) ...
Setting up libc6-dev:amd64 (2.19-18+deb8u10) ...
Setting up cpp-4.9 (4.9.2-10+deb8u1) ...
Setting up cpp (4:4.9.2-2) ...
Setting up libgcc-4.9-dev:amd64 (4.9.2-10+deb8u1) ...
Setting up gcc-4.9 (4.9.2-10+deb8u1) ...
Setting up gcc (4:4.9.2-2) ...
Setting up libstdc++-4.9-dev:amd64 (4.9.2-10+deb8u1) ...
Setting up g++-4.9 (4.9.2-10+deb8u1) ...
Setting up g++ (4:4.9.2-2) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up make (4.0-8.1) ...
Setting up libtimedate-perl (2.3000-2) ...
Setting up libdpkg-perl (1.17.27) ...
Setting up dpkg-dev (1.17.27) ...
Setting up build-essential (11.7) ...
Setting up openssl (1.0.1t-1+deb8u8) ...
Setting up ca-certificates (20141019+deb8u3) ...
Updating certificates in /etc/ssl/certs... 174 added, 0 removed; done.
Setting up curl (7.38.0-4+deb8u10) ...
Setting up libcurl4-gnutls-dev:amd64 (7.38.0-4+deb8u10) ...
Setting up sudo (1.8.10p3-1+deb8u5) ...
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Processing triggers for ca-certificates (20141019+deb8u3) ...
Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
Processing triggers for systemd (215-17+deb8u7) ...
 ---> 5c0310ae3ddc
Removing intermediate container f59911d3c822
Step 4/7 : RUN curl -L https://toolbelt.treasuredata.com/sh/install-debian-jessie-td-agent2.sh | sh
 ---> Running in e83c29c33e61
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0==============================
 td-agent Installation Script
==============================
This script requires superuser access to install apt packages.
You will be prompted for your password by sudo.
100   738  100   738    0     0   1449      0 --:--:-- --:--:-- --:--:--  1447
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3157  100  3157    0     0   105k      0 --:--:-- --:--:-- --:--:--  106k
OK
Get:1 http://packages.treasuredata.com jessie InRelease [1842 B]
Get:2 http://packages.treasuredata.com jessie/contrib amd64 Packages [399 B]
Hit http://security.debian.org jessie/updates InRelease
Get:3 http://security.debian.org jessie/updates/main amd64 Packages [644 kB]
Ign http://deb.debian.org jessie InRelease
Hit http://deb.debian.org jessie-updates InRelease
Hit http://deb.debian.org jessie Release.gpg
Hit http://deb.debian.org jessie Release
Get:4 http://deb.debian.org jessie-updates/main amd64 Packages [23.1 kB]
Get:5 http://deb.debian.org jessie/main amd64 Packages [9064 kB]
Fetched 9734 kB in 5s (1824 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  td-agent
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 55.0 MB of archives.
After this operation, 183 MB of additional disk space will be used.
Get:1 http://packages.treasuredata.com/2/debian/jessie/ jessie/contrib td-agent amd64 2.3.6-0 [55.0 MB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 55.0 MB in 2s (20.0 MB/s)
Selecting previously unselected package td-agent.
(Reading database ... 12532 files and directories currently installed.)
Preparing to unpack .../td-agent_2.3.6-0_amd64.deb ...
Unpacking td-agent (2.3.6-0) ...
Setting up td-agent (2.3.6-0) ...
Adding system user `td-agent' (UID 104) ...
Adding new group `td-agent' (GID 107) ...
Adding new user `td-agent' (UID 104) with group `td-agent' ...
Not creating home directory `/home/td-agent'.
Installing default conffile /etc/td-agent/td-agent.conf ...
THANK YOU FOR USING TREASURE DATA’s PACKAGED VERSION OF FLUENTD (TD-AGENT). IF YOU ARE NOT A TREASURE DATA CUSTOMER AND NEED HELP TROUBLESHOOTING OR THE ENTERPRISE VERSION VISIT https://fluentd.treasuredata.com
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for libc-bin (2.19-18+deb8u10) ...

Installation completed. Happy Logging!


 ---> 6bb3f47cc669
Removing intermediate container e83c29c33e61
Step 5/7 : RUN /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch fluent-plugin-record-reformer
 ---> Running in 4358c3476ca1
Successfully installed serverengine-2.0.6
Building native extensions.  This could take a while...
invalid options: -SHN
(invalid options are ignored)
Successfully installed strptime-0.2.3
Successfully installed dig_rb-1.0.1
Successfully installed fluentd-1.1.3
Successfully installed excon-0.62.0
Successfully installed multipart-post-2.0.0
Successfully installed faraday-0.14.0
Successfully installed elasticsearch-transport-6.0.2
Successfully installed elasticsearch-api-6.0.2
Successfully installed elasticsearch-6.0.2
Successfully installed fluent-plugin-elasticsearch-2.8.6
Parsing documentation for serverengine-2.0.6
Installing ri documentation for serverengine-2.0.6
Parsing documentation for strptime-0.2.3
Installing ri documentation for strptime-0.2.3
Parsing documentation for dig_rb-1.0.1
Installing ri documentation for dig_rb-1.0.1
Parsing documentation for fluentd-1.1.3
Installing ri documentation for fluentd-1.1.3
Parsing documentation for excon-0.62.0
Installing ri documentation for excon-0.62.0
Parsing documentation for multipart-post-2.0.0
Installing ri documentation for multipart-post-2.0.0
Parsing documentation for faraday-0.14.0
Installing ri documentation for faraday-0.14.0
Parsing documentation for elasticsearch-transport-6.0.2
Installing ri documentation for elasticsearch-transport-6.0.2
Parsing documentation for elasticsearch-api-6.0.2
Installing ri documentation for elasticsearch-api-6.0.2
Parsing documentation for elasticsearch-6.0.2
Installing ri documentation for elasticsearch-6.0.2
Parsing documentation for fluent-plugin-elasticsearch-2.8.6
Installing ri documentation for fluent-plugin-elasticsearch-2.8.6
Done installing documentation for serverengine, strptime, dig_rb, fluentd, excon, multipart-post, faraday, elasticsearch-transport, elasticsearch-api, elasticsearch, fluent-plugin-elasticsearch after 11 seconds
Successfully installed fluent-plugin-record-reformer-0.9.1
Parsing documentation for fluent-plugin-record-reformer-0.9.1
Installing ri documentation for fluent-plugin-record-reformer-0.9.1
Done installing documentation for fluent-plugin-record-reformer after 0 seconds
12 gems installed
 ---> 4db79640c552
Removing intermediate container 4358c3476ca1
Step 6/7 : EXPOSE 24224
 ---> Running in e09f02db180e
 ---> 75708fd35464
Removing intermediate container e09f02db180e
Step 7/7 : CMD exec td-agent -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT
 ---> Running in 927bdb731fc6
 ---> 9ff77d0acf21
Removing intermediate container 927bdb731fc6
Successfully built 9ff77d0acf21
Successfully tagged blogsample_fluentd:latest
WARNING: Image for service fluentd was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Building elasticsearch
Step 1/2 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.3
5.4.3: Pulling from elasticsearch/elasticsearch
d3aeceeb0289: Pull complete
8f71a54ee36c: Pull complete
71f1935b230b: Pull complete
16c3cc6647f0: Pull complete
c23ca9956682: Pull complete
fa98162d672f: Pull complete
34efa21beb0e: Pull complete
139c723cd9ea: Pull complete
03a95a5ee04e: Pull complete
7f59c15ccc56: Pull complete
Digest: sha256:b14e0b6d7819d9a755d380cf32641befadac1c3a96c48a69831005250cfe54fd
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:5.4.3
 ---> 2ae8547160a7
Step 2/2 : RUN cd /usr/share/elasticsearch &&   bin/elasticsearch-plugin install analysis-kuromoji &&   bin/elasticsearch-plugin install analysis-icu
 ---> Running in 79819937416c
-> Downloading analysis-kuromoji from elastic
[=================================================] 100%??
-> Installed analysis-kuromoji
-> Downloading analysis-icu from elastic
[=================================================] 100%??
-> Installed analysis-icu
 ---> 9f70a0c52685
Removing intermediate container 79819937416c
Successfully built 9f70a0c52685
Successfully tagged blogsample_elasticsearch:latest
WARNING: Image for service elasticsearch was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Pulling kibana (docker.elastic.co/kibana/kibana:5.4.3)...
5.4.3: Pulling from kibana/kibana
d3aeceeb0289: Already exists
8579720ccec7: Pull complete
047c80c6b8c9: Pull complete
dd3d39cdec23: Pull complete
360978fe41a9: Pull complete
35aa697b4267: Pull complete
166e29386d64: Pull complete
d7dd1d81bf77: Pull complete
ed50f0e6ccee: Pull complete
Digest: sha256:a14294558c0d4a03d0abf5264b8147fa2a00f9b090e48448a1586b075d14e4b1
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:5.4.3
Pulling nginx (nginx:1.11)...
1.11: Pulling from library/nginx
6d827a3ef358: Already exists
f8f2e0556751: Pull complete
5c9972dca3fd: Pull complete
451b9524cb06: Pull complete
Digest: sha256:e6693c20186f837fc393390135d8a598a96a833917917789d63766cab6c59582
Status: Downloaded newer image for nginx:1.11
Creating blogsample_nginx_1         ...
Creating blogsample_fluentd_1       ... error
Creating blogsample_elasticsearch_1 ...
Creating blogsample_fluentd_1       ...
Creating blogsample_go_1            ...
Creating blogsample_go_1            ... done
ERROR: for blogsample_fluentd_1  Cannot start service fluentd: driver failed programming eCreating blogsample_elasticsearch_1 ... done
Creating blogsample_kibana_1        ... done
Creating blogsample_nginx_1         ... done

ERROR: for fluentd  Cannot start service fluentd: driver failed programming external connectivity on endpoint blogsample_fluentd_1 (59456b19d7383bd270ed7424a55702be74a9b924c77c2db0275d6758d718bf32): Error starting userland proxy: listen tcp 0.0.0.0:24224: bind: address already in use
ERROR: Encountered errors while bringing up the project.

リソースが不足しているっぽい。

$ free -m
             total       used       free     shared    buffers     cached
Mem:          2001       1938         62          0          0         27
-/+ buffers/cache:       1910         90
Swap:            0          0          0

↓
$ free -m
             total       used       free     shared    buffers     cached
Mem:          2001       1084        917          0         12         81
-/+ buffers/cache:        990       1010
Swap:            0          0          0


## CPU
$ sar -u
12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
...
05:20:01 PM     all      0.04      0.00      0.03      0.01      0.02     99.90
05:30:01 PM     all      0.04      0.00      0.03      0.01      0.02     99.91
05:40:01 PM     all      0.04      0.00      0.03      0.01      0.02     99.90
05:50:01 PM     all     13.82      0.00      3.01      1.03      0.08     82.06
06:00:36 PM     all     14.26      0.00      5.28     78.84      1.12      0.50
06:10:01 PM     all     34.88      0.00      6.14     56.52      0.89      1.58
...


## ロードアベレージ
$ sar -q
12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
...
05:20:01 PM         0       147      0.00      0.00      0.00
05:30:01 PM         1       147      0.00      0.00      0.00
05:40:01 PM         0       147      0.00      0.00      0.00
05:50:01 PM         1       157      0.94      0.39      0.15
06:00:36 PM         0       277      9.69      7.82      4.20
06:10:01 PM         0       201      1.51      6.54      5.89
...


## メモリ
$ sar -r
12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
...

05:20:01 PM    909852   1139400     55.60    166756    410508    971816     47.42
05:30:01 PM    909884   1139368     55.60    166756    410508    971816     47.42
05:40:01 PM    909764   1139488     55.61    166756    410524    971816     47.42
05:50:01 PM     74796   1974456     96.35    234604   1085612   1069228     52.18
06:00:36 PM     65172   1984080     96.82       116     25968   3084236    150.51
06:10:01 PM    336440   1712812     83.58      4668     46448   2591224    126.45
...

## Disk I/O
$ sar -b

12:00:01 AM       tps      rtps      wtps   bread/s   bwrtn/s
...
05:20:01 PM      0.16      0.00      0.16      0.00      1.90
05:30:01 PM      0.15      0.00      0.15      0.00      1.70
05:40:01 PM      0.16      0.00      0.16      0.00      1.82
05:50:01 PM     89.59      6.19     83.40    330.35  12059.86
06:00:36 PM   1179.01   1157.77     21.24 110175.33   3395.56
06:10:01 PM    791.31    786.20      5.11  81238.48    647.16

CurlのSFTPにて、サーバにアクセスするためには...

--insecure にするといける。

$ curl --insecure -u sftp_user:pass sftp://domain/directory1/

MacからプリンターのIPを探す方法

dns-sdは、Macに接続しているデバイスIPアドレスを探すのに、便利

対応内容

1. dns-sdの対応しているサービスを表示

$ dns-sd  -B  _services._dns-sd._udp  local.

  Browsing for _services._dns-sd._udp.local.
  DATE: ---Fri 20 May 2016---
  23:54:37.797  ...STARTING...
  Timestamp     A/R Flags  if Domain  Service Type         Instance Name
  23:54:37.798  Add     3   4 .       _tcp.local.          _ssh
  23:54:37.798  Add     3   4 .       _tcp.local.          _sftp-ssh
  23:54:37.798  Add     3   4 .       _tcp.local.          _airplay
  23:54:37.798  Add     3   4 .       _tcp.local.          _raop
  23:54:37.798  Add     3   4 .       _tcp.local.          _googlecast
  23:54:37.798  Add     3   4 .       _tcp.local.          _nomachine
  23:54:37.798  Add     3   4 local.  _apple-mobdev2._tcp. _sub
  23:54:37.798  Add     3   4 .       _tcp.local.          _apple-mobdev2
  23:54:37.798  Add     3   4 .       _tcp.local.          _ipps
  23:54:37.798  Add     3   4 .       _tcp.local.          _ipp
  23:54:37.798  Add     3   4 .       _tcp.local.          _workstation
  23:54:37.798  Add     3   4 .       _tcp.local.          _http
  23:54:37.798  Add     3   4 .       _tcp.local.          _amzn-wplay
  23:54:37.798  Add     3   4 .       _tcp.local.          _rfb
  23:54:37.798  Add     3   4 .       _tcp.local.          _afpovertcp
  23:54:37.798  Add     3   4 .       _tcp.local.          _smb
  23:54:37.798  Add     3   4 .       _udp.local.          _net-assistant
  23:54:37.798  Add     2   4 .       _tcp.local.          _eppc
  ^C

2. デバイスインスタンス名を探す

$ dns-sd  -B  _ipp._tcp  local.

  Browsing for _ipp._tcp.local.
  DATE: ---Fri 20 May 2016---
  23:54:47.782  ...STARTING...
  Timestamp     A/R  Flags  if Domain  Service Type   Instance Name
  23:54:47.783  Add      3   4 local.  _ipp._tcp.     Officejet6500donna @ mbp
  23:54:47.783  Add      3   4 local.  _ipp._tcp.     raw2dir @ mbp
  23:54:47.783  Add      2   4 local.  _ipp._tcp.     Officejet 6600 @ mbp
  ^C

3. デバイスインスタンス名から、詳しいIP情報を取得

$ dns-sd  -L  "Officejet 6600 @ mbp"  _ipp._tcp  local.

  Lookup Officejet 6600 @ mbp._ipp._tcp.local.
  DATE: ---Fri 20 May 2016---
  23:54:55.252  ...STARTING...
  23:54:55.253  Officejet\0326600\032@\032mbp._ipp._tcp.local. can be reached at mbp2-2.local.:631 (interface 4)
   txtvers=1 qtotal=1 rp=printers/Officejet_6600 ty=Unknown \
    adminurl=https://mbp2-2.local.:631/printers/Officejet_6600 \
     note=Büro\ im\ Keller priority=0 product=\(Officejet\ 6600\ e-All-in-One\) \
      pdl=application/octet-stream,application/pdf,application/postscript,image/jpeg,image/png,image/pwg-raster \
       UUID=e7d11337-a440-3f2d-7168-b53de4325791 TLS=1.2 Color=T Scan=T \
        printer-state=3 printer-type=0x480900E
  ^C

4. IPアドレス(v4もv6も可)を取得

dns-sd -Gv4v6 mbp2-2.local

  DATE: ---Sat 21 May 2016---
  0:12:41.025  ...STARTING...
  Timestamp     A/R Flags if Hostname    Address                                      TTL
  0:12:41.025  Add  3  4 mbp2-2.local.   FE80:0000:0000:0000:AEBC:32FF:FEAE:CAEB%en0  120
  0:12:41.025  Add  3  4 mbp2-2.local.   FD00:0000:0000:0000:AEBC:32FF:FEAE:CAEB%  120
  0:12:41.025  Add  2  4 mbp2-2.local.   192.168.177.20                               120
  ^C

apple.stackexchange.com

djangoのtest(django-nose, coverage)を使用する時

djangoでtestをするとき、django-nose、coverageが使える。

$ pip install nose
$ pip install django-nose
$ pip install coverage
  • settigns.py
INSTALLED_APPS += ['django_nose', ]
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
NOSE_ARGS = [
    '--with-coverage',  # coverage を取る
    '--cover-html',
    '--cover-package=app.search,
    '--nocapture',
    '--nologcapture',
]

tests.py

import unittest
from django.test import Client

class SimpleTest(unittest.TestCase):
    def setUp(self):
        # Every test needs a client.
        self.client = Client()

    def test_details(self):
        # Issue a GET request.
        response = self.client.get('/customer/details/')

        # Check that the response is 200 OK.
        self.assertEqual(response.status_code, 200)

        # Check that the rendered context contains 5 customers.
        self.assertEqual(len(response.context['customers']), 5)

上記設定をした後に、以下を回す。

$ python apps/manage.py test app --settings=product.settings
$ tree cover/
cover/
├── coverage_html.js
├── index.html
├── app_views_py.html
├── jquery.hotkeys.js
├── jquery.isonscreen.js
├── jquery.min.js
├── jquery.tablesorter.min.js
├── keybd_closed.png
├── keybd_open.png
├── status.json
└── style.css

$ cd cover/
$ python -m http.server 9090

remotestance.com

21.22. http.server — HTTP サーバ — Python 3.6.5 ドキュメント

1億行のファイルを読み込む方法

ファイルを書き出す

  • manyrowstsv.py
#!/usr/bin/env python
# coding:utf-8

import signal
import sys
import os
import glob
import logging
import logging.handlers
import shutil
import tempfile
import random
import datetime
import string
import click
import itertools


def write_str_into_file(
    iterable,
    output_filename,
):
    with tempfile.NamedTemporaryFile(delete=False, dir='/var/tmp',) as f:
        for row in iterable:
            f.write(row)
        shutil.move(f.name, output_filename)
    if os.path.exists(f.name):
        os.remove(f.name)


class SignalException(Exception):
    def __init__(self, message):
        super(SignalException, self).__init__(message)


def do_exit(sig, stack):
    raise SignalException("Exiting")


class TsvRowGenerator(object):

    def __init__(
        self, dt_iso_max, dt_iso_min, date_iso_max, date_iso_min,
            ):
        self.dt_iso_max = datetime.datetime.strptime(
            dt_iso_max, '%Y/%m/%d %H:%M:%S')
        self.dt_iso_min = datetime.datetime.strptime(
            dt_iso_min, '%Y/%m/%d %H:%M:%S')
        self.date_iso_max = datetime.datetime.strptime(
            date_iso_max, '%Y/%m/%d')
        self.date_iso_min = datetime.datetime.strptime(
            date_iso_min, '%Y/%m/%d')
        delta = self.dt_iso_max - self.dt_iso_min
        date_delta = self.date_iso_max - self.date_iso_min
        self.int_delta = (delta.days * 24 * 60 * 60) + delta.seconds
        self.int_date_delta = (date_delta.days * 24 * 60 * 60) + \
            date_delta.seconds

    def iterows(self):
        yield (
            "\t".join(
                ["int", "short", "long", "double", "bool",
                    "char", "utf8", "dt_iso8601", "date_iso8601"]
                ) + "\n")
        while 1:
            rdp = random.randint(0, (1 << 32) - 1)
            random_second = rdp % self.int_delta
            randomtime = self.dt_iso_min + datetime.timedelta(
                seconds=random_second)
            random_date_second = rdp % self.int_date_delta
            randomdatetime = self.date_iso_min + datetime.timedelta(
                seconds=random_date_second)
            yield ("\t".join(
                [
                    str(rdp - (1 << 31)),
                    str((rdp >> 16) - (1 << 15)),
                    str(rdp - (1 << 31)),
                    str(random.uniform(0.1, 2.7)),
                    str(rdp % 2),
                    random.choice(
                        string.ascii_letters) + random.choice(
                        string.ascii_letters) + random.choice(
                        string.ascii_letters) + random.choice(
                        string.ascii_letters),
                    u"ごんた".encode('utf-8'),
                    randomtime.strftime("%Y-%m-%d %H:%M:%S"),
                    randomdatetime.strftime("%Y-%m-%d"),
                ]) + "\n")


@click.command()
@click.argument('rows', type=int, default=1000000)
@click.option(
    '-f', '--filename',
    default="~/kadai_1.tsv",
    )
@click.option('-D', '--dt-iso-max', default="2016/12/31 00:00:00")
@click.option('-d', '--dt-iso-min', default="2016/12/1 00:00:00")
@click.option('-T', '--date-iso-max', default="2016/12/31")
@click.option('-t', '--date-iso-min', default="2016/12/1")
def cmd(rows, filename, dt_iso_max, dt_iso_min,
        date_iso_max, date_iso_min):
    LOG_MANYROWSTSV = 'logging_warning.out'
    my_logger = logging.getLogger('MyLogger')
    my_logger.setLevel(logging.WARNING)
    handler = logging.handlers.RotatingFileHandler(
        LOG_MANYROWSTSV, maxBytes=2000, backupCount=5,)
    my_logger.addHandler(handler)
    s = datetime.datetime.now()
    print s + datetime.timedelta(0, 0, 0, 0, 0, 9)
    signal.signal(signal.SIGINT, do_exit)
    signal.signal(signal.SIGHUP, do_exit)
    signal.signal(signal.SIGTERM, do_exit)
    try:
        write_str_into_file(
            iterable=itertools.islice(
                TsvRowGenerator(
                    dt_iso_max, dt_iso_min, date_iso_max, date_iso_min,
                ).iterows(), rows + 1),
            output_filename=os.path.abspath(os.path.expanduser(filename)),)
        print os.path.abspath(os.path.expanduser(filename))
    except SignalException as e1:
        my_logger.warning('%s: %s' % (e1, datetime.datetime.now()))
        logfiles = glob.glob('%s*' % LOG_MANYROWSTSV)
        print logfiles
        sys.exit(1)
    finally:
        e = datetime.datetime.now()
        print str(e-s)


def main():
    cmd()


if __name__ == '__main__':
    main()
  • parselargetsv.py
#!/usr/bin/env python
# coding:utf-8

import signal
import sys
import os
import glob
import logging
import logging.handlers
import csv
import datetime
import click
import pickle
import struct
from manyrowstsv import write_str_into_file


class ParseRowsTsv(object):

    def __init__(
        self, file, inputf, outputf
            ):
        self.inputf = os.path.abspath(os.path.expanduser(inputf))
        self.outputf = os.path.abspath(os.path.expanduser(outputf))
        self.file = file

    def write_into_file(self):
        if self.file == 'pickle':
            write_str_into_file(self.pickle_tsv(), self.outputf)
        elif self.file == 'struct':
            write_str_into_file(self.struct_tsv(), self.outputf)

    def read_tsv(self):
        with open(self.inputf, "rb") as f:
            reader = csv.reader(f, delimiter="\t", lineterminator='\n')
            yield reader.next()
            for row in reader:
                row = (
                    int(row[0]),
                    int(row[1]),
                    int(row[2]),
                    float(row[3]),
                    int(row[4]),
                    row[5],
                    row[6],
                    row[7],
                    row[8],
                )
                yield row

    def pickle_tsv(self):
        for record in self.read_tsv():
            yield pickle.dumps(record)

    def struct_tsv(self):
        lines = self.read_tsv()
        line = lines.next()
        inits = struct.Struct(
            's '.join(
                [str(len(line[i])) for i in range(9)]) + 's')
        yield inits.pack(*line)
        for record in lines:
            s = struct.Struct(
                'i h l d ? %ds %ds %ds %ds' % (
                    len(record[5]), len(record[6]),
                    len(record[7]), len(record[8]),
                    )
                )
            yield s.pack(*record)


class SignalException(Exception):
    def __init__(self, message):
        super(SignalException, self).__init__(message)


def do_exit(sig, stack):
    raise SignalException("Exiting")


@click.command()
@click.option(
    '--file', type=click.Choice(['pickle', 'struct']),
    default='pickle')
@click.option('-i', '--inputf', default='~/kadai_1.tsv')
@click.option('-o', '--outputf', default='~/kadai_2.p')
def cmd(file, inputf, outputf):
    s = datetime.datetime.now()
    print s + datetime.timedelta(0, 0, 0, 0, 0, 9)
    # シグナル
    signal.signal(signal.SIGINT, do_exit)
    signal.signal(signal.SIGHUP, do_exit)
    signal.signal(signal.SIGTERM, do_exit)
    # ログハンドラーを設定する
    LOG_MANYROWSTSV = 'logging_warning.out'
    my_logger = logging.getLogger('MyLogger')
    my_logger.setLevel(logging.WARNING)
    handler = logging.handlers.RotatingFileHandler(
        LOG_MANYROWSTSV, maxBytes=2000, backupCount=5,)
    my_logger.addHandler(handler)

    parser = ParseRowsTsv(file, inputf, outputf)

    try:
        parser.write_into_file()

    except SignalException as e1:
        my_logger.warning('%s: %s' % (e1, datetime.datetime.now()))
        logfiles = glob.glob('%s*' % LOG_MANYROWSTSV)
        print logfiles
        sys.exit(1)
    finally:
        e = datetime.datetime.now()
        print str(e-s)


def main():
    cmd()


if __name__ == '__main__':
    main()

ファイルを読み込む

  • parsetsv_multitask_p3.py
#!/usr/bin/env python3
# coding:utf-8

import signal
import sys
import os
import glob
import logging
import logging.handlers
import datetime
import click
import pickle
import struct
import tempfile
import shutil
import math
import concurrent.futures
import errno


class SignalException(Exception):
    def __init__(self, message):
        super(SignalException, self).__init__(message)


def do_exit(sig, stack):
    raise SignalException("Exiting")


# ファイルを分割し、index、offset、offset + lengthを返す。
def tsv_separate_generator(inputf):
    CHUNK_SIZE = 1024 * 1024 * 100
    with open(inputf, 'rb') as f:
        f_size = os.stat(f.fileno()).st_size
        split_count = math.ceil(f_size / CHUNK_SIZE)
        start_offset = len(f.readline())
        for split_idx in range(split_count):
            offset = CHUNK_SIZE * (split_idx + 1) - 1
            f.seek(offset)
            last_line_len = len(f.readline())
            if offset < f_size:
                end_offset = offset + last_line_len
            else:
                end_offset = f_size
            yield (
                split_idx,
                start_offset,
                end_offset,
            )
            if end_offset >= f_size or last_line_len == 0:
                break
            start_offset = end_offset


def sum_file(self, files):
    with tempfile.NamedTemporaryFile(delete=False, dir='/var/tmp/',) as f:
        s = 0
        for file in self.files:
            with open(file) as f1:
                os.sendfile(f.fileno(), f1.fileno(), s)
            s += os.stat(file).st_size
        return f.name


class ReadTsvGenerator(object):

    def __init__(self, inputf, iterable):
        self.inputf = inputf
        self.iterable = iterable

    def read_tsv(self):
        with open(self.inputf, "rb") as f:
            start_offset = self.iterable[1],
            end_offset = self.iterable[2],
            f.seek(start_offset[0])
            start = start_offset[0]
            while start < end_offset[0]:
                row = f.readline()
                start += len(row)
                row = [
                    i.decode(
                        'utf-8'
                    ) for i in row.strip(b'\n').split(b'\t')
                    ]
                row = (
                    int(row[0]),
                    int(row[1]),
                    int(row[2]),
                    float(row[3]),
                    int(row[4]),
                    row[5],
                    row[6],
                    row[7],
                    row[8],
                )
                yield row


class ParseTsvGenerator(object):
    def __init__(self, iterable):
        self.iterable = iterable

    def pickle_tsv(self):
        lines = self.iterable
        next(lines)
        for record in lines:
            yield pickle.dumps(record)

    def struct_tsv(self):
        lines = self.iterable
        next(lines)
        for record in lines:
            s = struct.Struct(
                'i h l d ? %ds %ds %ds %ds' % (
                    len(record[5]), len(record[6]),
                    len(record[7]), len(record[8]),
                    )
                )
            yield s.pack(*record)


class ParseRowsTsv(object):

    def __init__(self, file, inputf, outputf):
        self.file = file
        self.inputf = os.path.abspath(os.path.expanduser(inputf))
        self.outputf = os.path.abspath(os.path.expanduser(outputf))

    # 単一タスク
    def dotask(self, rule):
        parsetsv = ParseTsvGenerator(
            ReadTsvGenerator(self.inputf, rule).read_tsv())
        if self.file == 'pickle':
            w = parsetsv.pickle_tsv()
        elif self.file == 'struct':
            w = parsetsv.struct_tsv()
        with tempfile.NamedTemporaryFile(
            delete=False, dir='/var/tmp', suffix='_dotask', prefix='tmp_',
                ) as f:
            for row in w:
                f.write(row)
            return f.name

    # マルチプロセス
    def multi_do_task(self):
        with concurrent.futures.ProcessPoolExecutor() as executor:
            future_to_tsv = {
                executor.submit(
                    self.dotask, rule
                ): rule for rule in tsv_separate_generator(self.inputf)}
            with tempfile.TemporaryDirectory(
                    suffix='_tsv', prefix='tmp_', dir='/var/tmp') as temp_dir:
                with tempfile.NamedTemporaryFile(
                        suffix='_tsv', prefix='tmp_',
                        delete=False, dir=temp_dir,) as f:
                    s = 0
                    for future in concurrent.futures.as_completed(
                            future_to_tsv):
                        chunk = future_to_tsv[future][2] - \
                            future_to_tsv[future][1]
                        with open(future.result()) as separatefile:
                            os.sendfile(
                                f.fileno(), separatefile.fileno(), s, chunk)
                            s += os.stat(separatefile.fileno()).st_size
                        try:
                            os.remove(separatefile.name)
                        except OSError as exc:
                            if exc.errno != errno.ENOENT:
                                raise
                    shutil.move(f.name, self.outputf)


@click.command()
@click.option(
    '--file', type=click.Choice(['pickle', 'struct']),
    default='pickle')
@click.option('-i', '--inputf', default='~/kadai_1.tsv')
@click.option('-o', '--outputf', default='~/zone/kadai_2v3.p')
def cmd(file, inputf, outputf):
    s = datetime.datetime.now()
    print(s + datetime.timedelta(0, 0, 0, 0, 0, 9))
    # シグナル
    signal.signal(signal.SIGINT, do_exit)
    signal.signal(signal.SIGHUP, do_exit)
    signal.signal(signal.SIGTERM, do_exit)
    # ログハンドラーを設定する
    LOG_MANYROWSTSV = 'logging_warning.out'
    my_logger = logging.getLogger('MyLogger')
    my_logger.setLevel(logging.WARNING)
    handler = logging.handlers.RotatingFileHandler(
        LOG_MANYROWSTSV, maxBytes=2000, backupCount=5,)
    my_logger.addHandler(handler)

    parser = ParseRowsTsv(file, inputf, outputf)

    try:
        parser.multi_do_task()

    except SignalException as e1:
        my_logger.warning('%s: %s' % (e1, datetime.datetime.now()))
        logfiles = glob.glob('%s*' % LOG_MANYROWSTSV)
        print(logfiles)
        sys.exit(1)
    finally:
        e = datetime.datetime.now()
        print(str(e-s))


def main():
    cmd()


if __name__ == '__main__':
    main()

Linuxのリソース確認にgrafanaとか言うのが大変便利

blog.serverworks.co.jp

最大流について

問題

s => t に最大量のデータを流す場合、最大どれだけのデータを送信できるか。

class MaxTraffic(object):
    def __init__(self, N=5):
        self.N = N
        self.edge = [[] for i in range(N)]
        self.used = [0 for i in range(N)]
        self.inf = 10 ** 9

    def append(self, _from, _to, cost):
        self.edge[_from].append(
            {
                'to': _to,
                'cap': cost,
                'rev': len(self.edge[_to]),
            }
        )
        self.edge[_to].append(
            {
                'to': _from,
                'cap': 0,
                'rev': len(self.edge[_from]) - 1,
            }
        )

    def dfs(self, v, t, f):
        """
        増加パスをDFSで探す
        """
        if v == t:
            return f
        self.used[v] = 1
        for i in range(len(self.edge[v])):
            e = self.edge[v][i]
            if not self.used[e['to']] and e['cap'] > 0:
                d = self.dfs(e['to'], t, min(f, e['cap']))
                if d > 0:
                    e['cap'] -= d
                    self.edge[e['to']][e['rev']]['cap'] += d
                    return d
        return 0

    def max_flow(self, s, t):
        flow = 0
        while 1:
            self.used = [0 for i in range(self.N)]
            f = self.dfs(s, t, self.inf)
            if f == 0:
                return flow
            flow += f

    def sample_append(self):
        self.append(0, 1, 10)
        self.append(0, 2, 2)
        self.append(1, 2, 6)
        self.append(1, 3, 6)
        self.append(2, 4, 5)
        self.append(3, 2, 3)
        self.append(3, 4, 8)
In [166]: m = MaxTraffic()
In [166]: m = MaxTraffic()

In [167]: m.sample_append()
In [167]: m.sample_append()

In [168]: m.max_flow(0, 4)
In [168]: m.max_flow(0, 4)
Out[168]: 11