Saturday Bazaar Review
22 July, 2018
Bazaar News & Events
As always, don't miss the chance to https://www.reddit.com/r/OpenBazaar/comments/90hiqp/shameless_self_promotion_and_requests_thread/ be a little shameless and promote your store
Fridays on Reddit in https://www.reddit.com/r/OpenBazaar/ /r/OpenBazaar !
If you're having an event in, around, or about OpenBazaar, let us know!
Get in touch on OpenBazaar, or find us on https://twitter.com/mazaclub_market Twitter @mazaclub_market
A couple of exciting developments we see this week
Brian tweeted that the upcoming integration with coinmarketcap's price ticker feed is nearly complete. Brian isn't one for release targets, so we know that if he's saying it's "nearly complete" it really is. We expect to see this in production soon.
In the openbazaar-go server we see a fresh release candidate for v0.12.1, which looks like it's about ready for the multicoin wallet upgrade. The openbazaar-desktop client also has a new release candidate for v2.2.2, and we can see there's been a good deal of work updating the interface to support the expanded cryptocurrency trading and the multicoin wallet.
Between those two main products, and the work we've seen on the various other components that will go into the multicoin wallet, it looks like this project is almost ready for the bravest of us to start testing.
We haven't seen many public commits to the http://openbazaar.com openbazaar.com repo, so we don't know how close the team might be to getting the web browsed based node working, but we did notice that there's been some new work on the upcoming widget. This is pretty exciting to us, as we'd really like to embed some of our openbazaar listings on other sites.
Next week, we'll join the openbazaar team on their development call to see what's in store for August.
Remote Server Error Troubleshooting
Last week we went through the process to back up your openbazaar node keys and data, and move those to a remote server running linux and docker.
We noted a serious issue with our nodes after we went to press. All of our items priced in fiat were not purchasable! We did some digging through the logs and found that openbazaard wasn't able to lookup DNS names for exchange rate tickers, so we went looking for the cause. We know that some of you are technical wizards, but a lot of you don't think you are. We'll show you how easy it can be to troubleshoot a malfunctioning openbazaar node running in a docker container, and we'll see two of our favorite causes for failure
DNS is our oldest favorite. We always check for DNS issues when we find problems with our servers. DNS issues can be the root cause of seemingly unrelated problems in a network or server.
Docker is our newest favorite. Docker and linux namespaces aren't always well understood, and we find the docker developers make some very odd choices in how containers operate. Docker errors can be hard to isolate, and even harder to find information on. You really need some google-fu to get docker working right sometimes.
First we noted that the openbazaar/server docker image is a bare-bones image, with nothing much inside besides the openbazaard binary. This leaves us without a direct way to look at what the problems are inside the container. We like to use docker-enter to open a shell inside, but there's no shell in the stock image.
We grabbed a copy of a network toolbox that guruvan publishes, and tested the networking functionality in the container
docker run -it --rm --net container openbazaar_testnode-btc_1 guruvan/network-tools
This gives us a gentoo image with a few tools to test the network like dig & tcpdump, and a standard bash shell
# ping http://google.com google.com
PING google.com(sea15s01-in-x0e.1e100.net (2607:f8b0:400a:806::200e)) 56 data bytes
64 bytes from sea15s01-in-x0e.1e100.net (2607:f8b0:400a:806::200e): icmp_seq=1 ttl=57 time=0.660 ms
# ping ticker.openbazaar.org
PING ticker.openbazaar.org (18.104.22.168) 56(84) bytes of data.
64 bytes from 22.214.171.124 (126.96.36.199): icmp_seq=1 ttl=53 time=18.9 ms
This was perplexing. We expected the container to not resolve DNS properly, but it is. We did some extensive testing at this point, to make sure that none of the options we've used to configure our remote nodes were the cause. We checked for operational differences between several different versions of the openbazaar-go code, differences between running a bitcoin ob node and a bitcoincash ob node, and we tried turning SSL on and off.
We also wanted to verify that the error was occurring whenever openbazaard wanted to look up a name, so we tested out the email integration feature. We set up an account in our Google Apps hosted email service, and gave it a go. To our dismay, it didn't appear that the server even tried to find http://smtp.gmail.com smtp.gmail.com , and simply failed immediately.
Ever curious, we decided it's time to dig out a copy of tcpdump and investigate the traffic coming from our container, to see if we could see any DNS requests.
docker run -it --rm --net container openbazaar_testnode-btc_1 guruvan/network-tools
# tcpdump -ni eth0 port 53
Knowing that we could hosts by name from the command line, and seeing that openbazaard wasn't even trying to look up names we needed to test openbazaard a little further.
We wanted to rule out the possibility that there was something wrong with the openbazaard that we had, so we copied it out from the docker image onto our nixos host, and ran it using the same node directory we were testing with above.
docker-compose up -d testnode-btc
docker-compose cp /opt/openbazaard /var/lib/openbazaar
this puts a copy of openbazaard in our node's directory. We check the config for openbazaar, and the ports it's using, to make sure those ports are available on our host
# netstat -an | grep LIST
tcp6 0 0 :::4001 :::* LISTEN
tcp6 0 0 :::4002 :::* LISTEN
tcp6 0 0 :::9005 :::* LISTEN
will show you what ports are listening, the default openbazaar ports are the ones above
./testnode-btc/obdata/openbazaard start -d ./testnode-btc/obdata -l debug
14:52:15.789 [fetch] [ERROR] Failed to fetch from https://ticker.openbazaar.org/api Get https://ticker.openbazaar.org/api: dial tcp: lookup ticker.openbazaar.org: no such host
We're looking for the errors we saw when the node tried to check exchange rates. We're not seeing those, so we opened up the client, connected to our testnode and tested our email integration connection. Success!
So now we knew that our issue was isolated to openbazaard, but only when running inside docker, but the docker containers appeared to work correctly from the command line.
Finally, we had to do our best to determine whether the culprit was openbazaard or docker, but nothing appeared to be wrong with docker so we checked in with the openbazaar development team on github
guruvan filed this issue for us https://github.com/OpenBazaar/openbazaar-go/issues/1023 https://github.com/OpenBazaar/openbazaar-go/issues/1023
explaining what the servers were doing, and explaining what we'd done to isolate the cause. Sadly, after a short wait for a response, there were no clues to be had! The openbazaar team hadn't seen this issue on any of their own servers!
After double checking that we didn't have any special configuration options set in our docker daemons, we knew this was a docker issue. guruvan's been working with docker for over 4 years, and says he's seen some exceptionally stupid choices made by the docker team, so he hunted one down. Sure enough, there's a 2 year old docker forum post with a likely looking issue. An obscure option set by docker in the container prevents some applications from looking up names.
At this point we knew we were going to have to tinker around inside the container, and we'd be best off making a new docker image with a command line shell we could use
We wanted to keep everything as much the same our original openbazaar/server:v0.12.0, so we changed the base image from "scratch" to "alpine:latest" which is a tiny Alpine Linux image. We run the openbazaar image we create like this, so we can look more carefully
docker build -f Dockerfile.new -t mazaclub/openbazaar:v0.12.0-alpine .
docker run -ti --rm \
-p 4001:4001 \
-p 4002:4002 \
-p 9005:9005 \
-v /home/username/openbazaar/testnode-btc/obdata:/var/lib/openbazaar \
--entrypoint /bin/ash \
Now we can look around and see what docker is doing inside this container
# cat /etc/resolv.conf
options edns0 ndots:0
AAAHA. We should have known all along that it was something docker was doing. We wanted to try to get docker's dns all the way out of the way for testing, and we know many of you don't know how to use ```vi``` very well.
# echo "nameserver 188.8.131.52" > /etc/resolv.conf
We checked our openbazaard with the client app, and tested the email integration again. Success. With a little more testing we determined that openbazaard would only function correctly if the resolv.conf was corrected before starting openbazaard!
Since this isn't how the docker images we're using are built, we had to make some changes to make this work. We needed a script that would fix our resolv.conf, and then start openbazaard for us, with all the options we've set in our docker-compose.yml
We used this ob://QmY8HpD6iLHqZUErP6PZhW9QGiqFehFRBbAGVLcA34mxdk/store/docs-ob-server-dockerfile.mazaclub Dockerfile and this ob://QmY8HpD6iLHqZUErP6PZhW9QGiqFehFRBbAGVLcA34mxdk/store/docs-ob-server-entrypoint.sh entrypoint script to get the job done
docker build -f Dockerfile.mazaclub -t mazaclub/openbazaar-go:test1 .
we tested it by changing the "image:" line in our docker-compose.yml for our testnode and ran
docker-compose up -d testnode-btc ; tail -f testnode-btc/obdata/logs/ob.log
No errors! we checked to make sure we saw the correct /etc/resolv.conf
docker-compose exec testnode-btc cat /etc/resolv.conf
We connected to the node with our client, tested our email integration. Still successful, we proceeded on to verify our solution was working buy setting up some test listings, pricing them in fiat, and buying them. With everything looking about right, we changed a few more nodes to use the new docker image and script, and made sure they could all purchase fiat priced items from each other.
With everything working correctly, it's time to publish our changes, and offer to file a Pull Request on github for the openbazaar team. So, like with any project, we forked their repo into our mazaclub account, added our new Dockerfile and script, and pushed it up to github.
We still have one issue that we're following up on this week. Why isn't the OB1 team seeing this on their servers? We didn't ask what OS they're running their nodes on, but we're guessing that it's ubuntu or another debian based system, and this issue is fixed in the maintained packages. We run most of our servers with nixos, and know the nix team probably hasn't patched the docker sources. We'll check out a ubuntu server this week, and see if we have to use our updated docker image, or not.
When we do that, we'll update our Docs: series in our mazaclub publications store with a distilled how-to move to a remote server, with exact step-by-step instructions on ubuntu.
You can find an https://hub.docker.com/r/mazaclub/openbazaar-go/ automated dockerhub build of the image we're using.
mazaclub/openbazaar-go will always be the latest stable version
docker pull mazaclub/openbazaar-go
Last week we also covered the basic backup process you need to keep your openbazaar node data safe. We backed up the seed for our wallet and ob identity, and made a tarball of the node data, which includes our listings, about page, avatar, and so on.
This week we took a quick look through the restore command offered by openbazaard, and tried a few failure modes.
We're pleased to say that the command line tool was able to restore our wallet, and our openbazaar node's identity key, but we weren't able to take advantage of the command's ability to download our store's data from other nodes who still might have copies of it.
We didn't want to try this on one of our production stores, so we setup a test store. We made a basic profile, created a couple of listings, took the store down and backed it up. We modified those listings, connected to a few of our other nodes, and followed our test store, hoping that our other nodes would "pin" the ipfs data like our listings and profile information.
Once we did this, we let all the nodes sit for a little while (a couple hours) and do their thing on the network. We then took our testnode down, and deleted our data directory. We used tar to restore the data directory we backed up when we started, and then we tried the openbazaard restore command. While it appeared to operate correctly, we were a little disappointed. When we accessed the node from our client, we found that the restore had not downloaded our modified profile and listings. Though our node-identity and wallet were intact as expected, we had our original profile, and listings.
We know it takes time for network nodes to connect to each other, and copy relevant data, so we decided we'd take a more thorough look at this again after our test node has some time to simulate a real store. We'll give this a week, and try again.
In the meantime, we suggest making regular backups of your node's data directory, so that you always have your latest listings saved. Your store is available to buyers even while it's offline, so there's no reason not to make backups after you update your store, and make regular backups even if you don't make updates regularly.
A Day Out of Time
That's what we need to catch up! OpenBazaar is a complex collection of components, and, as you can see from our troubleshooting exercise this week, it can take a long time to sort out why things work, or don't work correctly. A lot of work has gone into making OpenBazaar an open and free platform and we like to take the time to be thorough as we understand how it works, and how it can be used.
Have something you'd like to see published in the Bazaar Review? Get in touch with us on OpenBazaar, or send us a message on Twitter https://twitter.com/mazaclub_market @mazaclub_market - our DMs are always open!
We planned these this week, but schedules didn't permit us to get them published. We have several technical reviews planned for the next weeks, as we explore all the openbazaar features and code.
- email notifications - we finally have ours working, we'll run through a typical GoogleApps configuration.
- remote node additional security measures
- How search works with multiple providers on the ob network
- a look at our nodes running the latest greatest code you shouldn't run yet
See our listings for more information on advertising your store in mazaclub's Saturday Bazaar Review and Sunday Bazaar Shopper.
|Store||mazaclub - BTC|
|Tags||article, blog, openbazaar, saturday-bazaar-review, news|
|Categories||Articles And Podcasts|
|Open||Listing with OpenBazaar|