We use terraform a lot at work and reading some of the terraform configuration or modules and the huge number of variables that sometimes exist, I start to question whether I should just use the underlying resource directly instead of the module abstraction. It comes from the need to paramaterise everything. You get so many variables that it seems like you’ve built another domain specific type that is more complex than the pieces you’re abstracting. ...
Git - committing as multiple identities (personal and work email)
The problem is pretty straight forward: I have a personal email and a work email and depending on the project I’m working on, I should commit as the identity relevant to that repo. So in a personal project, the git author should be damien@personalemail.com and within a work project, the git author should be damien@workemail.com. Git config has the ability to include other config files based on the directory the repo is in outline in the git-config Includes. Let’s assume our ~/.gitconfig file looks something like this: ...
Azure functions as static site host
Azure functions can be very cheap and very easy to manage. Hosting a single page app (SPA) that does very few requests to the host and is basically just a few files to be delivered to the client seems perfect for using the consumption pricing of azure functions. Basically only pay for a request vs a monthly fee. Azure Resources We’ll need a few resources in Azure, they are: Azure Function app Azure storage account with a blob container that has public access These are very straight forward to create in azure and plenty of articles if you haven’t done this before. ...
Azure GPUs, Jupyter and Machine Learning
I’m a big advocate of the cloud and it’s ability to provide just enough resources ad hoc. You can use whatever you want, and pay for it just when using it. In machine learning there are services such as Google’s ML Engine or Azure’s upcoming Batch AI but during development, data preprocessing etc sometimes you want immediate iterative processes. In these cases, you can’t go past a Jupyter notebook and in this case, running that on a VM. In this post, I’ll outline how I’ve setup such an environment in Azure, focusing on ability to build it up and tear it down via the CLI and using a cheaper VM during development and an easy jumpt to then being able to run it on a GPU machine once things are running smoothly. ...
Convert and using the MNIST dataset as TFRecords
TFRecords are TensorFlow’s native binary data format and is the recommended way to store your data for streaming data. Using the TFRecordReader is also a very convenient way to subsequently get these records into your model. The data We will use the well known MNIST dataset for handwritten digit recognition as a sample. This is easily retrieved from tensorflow via: from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets( "/tmp/tensorflow/mnist/input_data", reshape=False ) We then have mnist.validation, mnist.train and mnist.test data sets. ...
Getting started with PowerShell and Azure on macOS
PowerShell 6.0 has now become generally available and with it the ability to use PowerShell cross platform. With all of its goodness, I was most excited about Azure powershell cross platform capabilities that this brings about. Firstly, the azure-cli is also a great cross platform tool, but sometimes the powershell capabilities seem to outshine this CLI. Installing PowerShell on macOS The installation instructions on the repo are pretty straight forward, but for summary: ...
Getting started with TensorFlow in Google ML Engine
In 2017, there seems no doubt that if you aren’t running your ML training on a GPU you just aren’t doing things right ?. At home, my only computer is my MacBook Pro which is great to develop on, but would take an extremely long time to train something such as an image classification task. Saying this, I’d love to have a GPU machine at home, but I also love the opportunity to use this hardware in the cloud without having to power it, upgrade it and generally take care of something you actually own. Thus I am lead to investigate using Google’s ML Engine to train my models. ...
Ghost from an NPM module and hosted in Azure
It was great to see you can now use Ghost as an NPM module. The upgrade process used to be a pain prior to the 1.0 release. Saying this, I ran into a few bumps trying to host this on Azure. Follow the docs on Ghost as an NPM module to get started Set server port at runtime Node runs behind IIS and is reverse proxied so the port we are assigned is not static. In your index.js as setup in 1. add a line to set the port from the environment variable, so your ghost server start lines are as so: ...
Image classification using TensorFlow estimators and TensorFlow Hub for transfer learning
This notebook is available as a codelab TensorFlow Hub was announced at TensorFlow Dev Summit 2018 and promises to reduce the effort required to use existing machine learning models and weights in your own custom model. From the overview page TensorFlow Hub is a library to foster the publication, discovery, and consumption of reusable parts of machine learning models. A module is a self-contained piece of a TensorFlow graph, along with its weights and assets, that can be reused across different tasks in a process known as transfer learning. ...
Images with directories as labels for Tensorflow data
A common format for storing images and labels is a tree directory structure with the data directory containing a set of directories named by their label and each containing samples for said label. Often transfer learning that is used for image classification may provide data in this structure. Update May 2018: If you would like an approach that doesn’t prepare into TFRecords, utilising tf.data and reading directly from disk, I have done this in when making the input function for my Dogs vs Cats transfer learning classifier. ...