-
Notifications
You must be signed in to change notification settings - Fork 78
add a Docker image with benchmarking utilities #44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| FROM golang:1.12.1-alpine3.9 AS goycsb | ||
|
|
||
| RUN apk add git build-base \ | ||
| && git clone https://github.com/pingcap/go-ycsb $GOPATH/src/github.com/pingcap/go-ycsb \ | ||
| && cd $GOPATH/src/github.com/pingcap/go-ycsb \ | ||
| && make | ||
|
|
||
| FROM alpine:3.9 | ||
| MAINTAINER PingCAP <cloud@pingcap.com> | ||
|
|
||
| COPY --from=pingcap/tidb-sysbench /usr/local/bin/sysbench /usr/local/bin/sysbench | ||
| COPY --from=pingcap/tidb-sysbench /usr/local/share/sysbench /usr/local/share/sysbench | ||
|
|
||
| COPY --from=dmonakhov/alpine-fio /usr/local/bin/fio /usr/local/bin/fio | ||
|
|
||
| COPY --from=goycsb /go/src/github.com/pingcap/go-ycsb/bin/go-ycsb /go-ycsb/bin/go-ycsb | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also does this work in alpine? And maybe you can copy from pingcap/go-ycsb directly instead of build the go-ycsb yourself.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I haven't tested this, but it should work. Copying from pingcap/go-ycsb creates additional fragility unless we define a contract for go-ycsb (linux distro, binary location, static build)
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The go-ycsb repo already has a Dockerfile. It also uses alpine as builder and final image. |
||
| COPY --from=goycsb /go/src/github.com/pingcap/go-ycsb/workloads /go-ycsb/workloads | ||
| COPY --from=goycsb /go/src/github.com/pingcap/go-ycsb/tool /go-ycsb/tool | ||
|
|
||
| ADD README.md /README.md | ||
|
|
||
| CMD ["bash"] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| This image includes various utilities used to benchmark TiDB. | ||
|
|
||
| # Performance tests | ||
|
|
||
| * sysbench | ||
| * go-ycsb | ||
|
|
||
|
|
||
| # Disk tests | ||
|
|
||
| *Note:* All the commands need to be run under the disk mount point. And the test.data file need to be deleted before test. | ||
|
|
||
| ## fio | ||
| * Read | ||
| ``` | ||
| fio -ioengine=libaio -bs=32k -direct=1 -thread -rw=read -size=10G -filename=test.data -name="PingCAP max throughput" -iodepth=4 -runtime=60 | ||
| ``` | ||
|
|
||
| * Write | ||
| ``` | ||
| fio -ioengine=libaio -bs=32k -direct=1 -thread -rw=write -size=10G -filename=test.data -name="PingCAP max throughput" -iodepth=4 -runtime=60 | ||
| ``` | ||
|
|
||
| * Random read | ||
| ``` | ||
| fio -ioengine=libaio -bs=32k -direct=1 -thread -rw=randread -size=10G -filename=test.data -name="PingCAP max throughput" -iodepth=4 -runtime=60 | ||
| ``` | ||
|
|
||
| * Random write | ||
| ``` | ||
| fio -ioengine=libaio -bs=32k -direct=1 -thread -rw=randwrite -size=10G -filename=test.data -name="PingCAP max throughput" -iodepth=4 -runtime=60 | ||
| ``` | ||
|
|
||
| * Random read & write | ||
| ``` | ||
| fio -ioengine=libaio -bs=32k -direct=1 -thread -rw=randrw -percentage_random=100,0 -size=10G -filename=test.data -name="PingCAP max throughput" -iodepth=4 -runtime=60 | ||
| ``` | ||
|
|
||
| ## dd | ||
|
|
||
| * Cache write | ||
| ``` | ||
| dd bs=4k count=400000 if=/dev/zero of=test.data | ||
| ``` | ||
|
|
||
| * Direct write | ||
| ``` | ||
| dd bs=4k count=400000 if=/dev/zero of=test.data oflag=direct | ||
| ``` | ||
|
|
||
| * Direct read | ||
| ``` | ||
| dd bs=4k count=400000 if=test.data of=/dev/null& oflag=direct | ||
| ``` |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| pushd ../tidb-sysbench | ||
| ./build.sh | ||
| popd | ||
| docker build -t pingcap/tidb-bench . |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| FROM alpine:3.9 | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we need to make this a separate image?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes, because we patch sysbench. If we can get our patch upstream it won't really be needed. |
||
| MAINTAINER PingCAP <cloud@pingcap.com> | ||
|
|
||
| # Install sysbench | ||
| # gcc and mariadb-dev install shared libraries used at runtime | ||
| RUN apk add gcc mariadb-dev | ||
|
|
||
| RUN apk add --virtual .build-deps git build-base automake autoconf libtool --update \ | ||
| && git clone https://github.com/akopytov/sysbench.git \ | ||
| && cd sysbench \ | ||
| && git checkout 1.0.17 \ | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Make the version as an environment variable?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. okay |
||
| && ./autogen.sh \ | ||
| && ./configure --disable-shared \ | ||
| && make \ | ||
| && make install \ | ||
| && cd .. \ | ||
| && rm -r sysbench \ | ||
| && apk del .build-deps | ||
|
|
||
| # TiDB patch for sysbench data loading (prepare) | ||
| RUN apk add --virtual .build-deps wget \ | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why run
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it is easy to |
||
| && wget https://raw.githubusercontent.com/pingcap/tidb-bench/master/sysbench-patch/oltp_common.lua \ | ||
| && apk del .build-deps \ | ||
| && mv oltp_common.lua /usr/local/share/sysbench/oltp_common.lua \ | ||
| && chmod +x /usr/local/share/sysbench/oltp_common.lua | ||
|
|
||
| RUN apk add mysql-client | ||
| RUN apk add bash | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why add bash in this docker image? The tidb bench script is not included in this image.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think I agree that these last two can be left out and put in tidb-bench so that this image is just the patched sysbench. It just depends if you want to attach to the container for debugging. |
||
|
|
||
| WORKDIR "/usr/local/share/sysbench" | ||
| CMD ["sysbench"] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| docker build -t pingcap/tidb-sysbench . |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we just copy this binary to make it work? Is the fio a static binary or dynamically linked binary? From its Dockerfile I don't see any special instruction for static build.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't tested this yet.