Taras Mankovski Blog » Beecoop

Architecture of our Infrastructure

Taras — Wed, 30 Sep 2009 20:29:16 +0000

My main goal is to create a system that will scale easily and allow us to use commodity infrastructure elements. These commodity infrastructure elements each have their advantages and disadvantages. The architecture of the system aims to utilize the advantages and minimize the disadvantages.Here is a list different functions performed by our system.

Public User Interface
Collection
Processing
Storing

Public User Interface

Running on Cherokee Web Server. Cherokee is a high efficiency web server that has low requirements and can handle very high loads. This element can be served sufficiently well by an inexpensive virtualized web server running in the cloud.

Collection

Collection element of the system will be performed by scrapy spiders which will get their tasks from the Task Server and store the result in the Beecoop Data Store.

Processing

Processing is done by workers that, like Collection, get their tasks from the Task Server and either store their data in Beecoop Data Store or PUI database.

Both Processing and Collection require a lot of RAM and CPU. We’re going to attempt to use commodity hardware to fulfill these functions. Core Networks has inexpensive hardware that we can try to use. If this fails then we can use Amazon EC2.

Storing

The collection element will capture a lot of different kinds of data. In our testing, 1 week of execution, we collected over 500,000 units of data. To handle this volume of information we’re going to utilize Google Data Store, because it will give us a huge database that we will not have to scale ourselves.

Visual Representation of our Infrastructure

Building python development environment

Taras — Wed, 30 Sep 2009 15:30:52 +0000

I’m preparing to start development on beecoop and anonymous websites and I would like to establish a comfortable python development environment to make development and deployment of different applications easier.

There are several factors that I need to consider:

I’m going to be working on several different projects at the same time.
Each project is going to be using different python modules.
Some projects will run on apache, others on cherokee.
Cherokee does not have installer for Mac OS X, so I can not replicate our production environment locally.
I would like to start working in TDD
Last and not least, I have to make it easy for our sysadmin to deploy these projects.

Here are technologies that I’ve found so far and how I would like to combine them together.

Virtualenv with –no-site-packages switch allows you to create virtual environments that are seperate from your system. This forces you to be conscious of what packages your application is using. This is particularly useful when you’re doing development of applications that require different versions of python. We’re going to stick to python2.5 for the near future so this is not a major concern for us.

Buildout is the tool that i think we’re actually going to be using. It makes it easy for sysadmin to deploy the applications, while maintaining the benefit of forcing us to be conscious of what packages particular application requires. Buildout makes virtualenv unnecessary for us because it simulates virtualenv’s functionality by including the paths for all necessary packages in sys.path of every project.

Paste Script allows to create packages that we can include inside of our applications. It makes it easy to integrate multiple modules that are under development into our applications while keeping things clean and modular.

Visual Representation of Business Processes

Taras — Wed, 30 Sep 2009 15:30:24 +0000

I’m currently working on Beecoop admin tool that would allow me to create trac projects quickly and easily.

I’d like to write about Beecoop admin, but that’s not the point here.

In process of looking for information on iPython parallel processing, I came across a tutorial on how to visually manage processes with Vision.Vision is the tool that the presenter is using.

According to Vision website – “Vision is a visual-programming environment in which a user can interactively build networks describing novel combinations of computational methods, and yielding new visualizations of their data without actually writing code.”

Here is how I read this:
Vision is a visual-programming environment in which a user can interactively build data processing applications from reusable pieces of code.

Vision was created to manipulate molecule modes, but I think the applications are vast. For example, you could use it to manipulate information in multiple rss feeds, very similar to how Yahoo Pipes works, but you can also write your own processing functions in python.

It would be very cool to try to represent visually business processes. Once this is done, expend on these processes by implementing hooks into APIs of various web services.

I need to think about this later and I’ll write part 2 to this post later.

Back to work now,
Taras

Day 1: Testing Process

Taras — Tue, 05 Aug 2008 05:00:06 +0000

The first day of our Process week has been interesting and somewhat productive. We started off by taking over one of the conference rooms. We setup my laptop in front of the white board and recorded our conversation. We talked for about 4 hours. Here is a summary of this conversation.

First of all, we need to change how we implement projects because the process that we have right now causes too many revision cycles. From our experience, revision cycles are very distracting and time consuming. Sometimes each revision cycles takes from 2 days to 1 week.

Here is what a typical revision cycle looks like:

We implement all of the features in the spec and pass the site back to client advisor
Client Advisor either does testing himself or passes on to the Client to do testing and find bugs
Client or Client Advisor (depending on who is doing the testing) creates a list of bugs
Client Advisor passes the bugs to us
We fix the bugs and go back to step 1

The real problem is that we leave it up to the Client or Client Advisor to find bugs in the website. This adds work to Client Advisor and creates a perception of a faulty product. We would like to eliminate this as much as possible.

I think one way to do it, is to take a more proactive approach to testing. I think instead of testing at the end, we should plan testing at the beginning. What this means is that we would actually create tests for features before the feature even goes to the developer. These tests should be provided to the developer with the tickets for the feature.

This is good for several reasons:

It gives a developer better understanding of how the feature is going to be used.
It allows to predict issues that might come up doing the development process.

So how would we actually do this?

Let’s take a Standard Art Gallery site as an example (these are the kind of sites that Artgal sells).

A Standard Art Gallery Site has the following features:

Frontend

Home Page – usually some text, a featured artwork, list of categories
Artists Page – alphabetical list of all artists
Artwork Page – list of all artwork sorted by last name of the author then by title of the artwork
Category View – to display artwork in specific category
About Us – static content page
Contact Us – contact form

Frontend Admin

Edit Text on Home Page
Add/Edit/Delete Artwork
Add/Edit/Delete Artist
Edit description on Category view
Edit About Us Page
Edit Contact Us Page

Now, let’s take a look at how we might actually build tests for some of these features.

First of all our goal is to make the implementation of the site as simple as possible. To do this, we have to make sure that we have predefined wireframes for this kind of site. Doing this will show the developer how the views should be setup. What we don’t want is for the developer to waste time creating the user interface.

Let’s assume that the developer has good detailed tickets that allow him/her to implement the feature according to spec. What should the test look like?

Let’s take “Exit Text on Home Page” as an example. Here is what a test might look like.

Go to Home Page
Client login at the bottom right
Enter “admin” username & password
You should now be redirected back to home page
Click on edit beside the main text on home page
You should see a form that allows you to edit that text.
Change the text to “this is a very long text …. “
Click save
You should get “Your Text has been changed”

Each step has to be completed as per description for the test to pass.

For the developer, this allows to follow step by step and know exactly what’s going to be expected of his implementation and how it should be implemented.

For the tester, it gives instruction on how to test this features.

This can also be used as documentation for the client.

So it’s a win-win-win for everyone. And should potentially allow us to catch most of the bugs before the site goes to the client.

Next step is to see what technology we can use to automate this process.

I would love to hear your feedback,

Let me know what you think.

We need a new term to describe us.

Taras — Sun, 03 Aug 2008 20:44:08 +0000

I’ve been trying to find a word to unify all of us. When I say “us” I mean all of the people who contribute to creating the web. This includes developers, designers, copy writers, internet marketers, information architects, client advisors. The people who make the web possible.

Trevor mentioned to me once he believes that Web Developers of today are like the masons of the medieval period. These people would work on different projects like houses, churches and castles. They all had different skills and were engaged as their skills were necessary.

I believe that this resembles how the web is built today. We all work on projects together and bring expertise to the project.

Now the question is what do we call ourselves? The problem for me is that I often refer to all people involved in creating websites, so my sentences usually end up looking something like “I work with developers, designers, copy writers and marketing people.” There’s got to be a better way to refer to everyone together in a single word.

The other thing that I think we should consider in choosing a word is the fact that there are 2 types of web people: employed and self-employed or freelancers. So it would be good to have 2 variations of this word that would communicate their employment status.

So if you have any ideas or suggestions,
I’m all ears.

I’m looking forward to your feedback.

Hosted Development Environments

Taras — Wed, 16 Jul 2008 04:22:56 +0000

In process of building projects I am beginning to notice some patterns emerging in the requirements that we have for development environments. I think in most cases we need very similar features. I would love to be able to create a service that makes it easy for us work on projects together.

Here is a list of things that I would like to see this service do.

Easy to add and create development environments
Development environment must be pre configured for the type of development that is going to be done (this is important because Joomla, Drupal, Wordpress development is slightly different – same for most part, but Python is very different)
It must have an api that allows us to manage permissions on each individual development environment quickly and efficiently
It must be secure, giving access to source code only by people involved in the project
Must allow SSH access
Must use Public Keys for authentication to make authentication quicker and easier (this applies to both ssh and mysql)
Must replica-table on many different servers – this will allow us to create a grid of computers working together to server individual development environments

Some necessary tools

svn, git, baazar
screen
xdebug for php remote debugging

These are my thoughts. I’m going to be changing them around, but for now here they are

Mid July to End of August 2008

Taras — Wed, 16 Jul 2008 04:13:17 +0000

The next month and a half are going to be very interesting.

We’ve decided to take a break from doing projects to focus on our process. We would like to really streamline our operation. We’ve been popping out projects and things have been very interesting but we would like to introduce more efficiency.

Our goals are:

Complete my transition from Lead Developer to Architect
Make it easy for me to be hands of on projects and empower our team to implement projects quicker and more efficiently
Build tools to make development process easier
Create Revision process to steamline process of getting change requests to developers
Setup testing & QA process for projects
Create process to make it easier for Client Advisors to track and get updates on projects

Some of the ideas that we have for the next 6 weeks are

Begin discussion with Client Advisors on tools that they need to make communication and work with us easier
Work with developers to create a process to simplify transfer of requirements
Begin work with Margo on building branding identity for osCandy, Positive Sum and Beecoop
Travel to Moldova to spend time with developers to make tools to make process of site building easier (for developers and client advisors – different tools)

Also during this period we intend on finishing all projects that we have outstanding. This will allow us to start September off a clean slate. No more old projects, everything starting september will be clean and new.

Also, we’re looking for a loft. We would like it to be a place were both Josh and I can live. Our main goal is to allow developers, designers, client advisors and project managers to drop in at any time and chill out and get some work done. I’d love to be able to bring people together in this space. I think it will be a great place for josh and I to focus on what we have ahead of us. Beecoop is our future and I’d love to be able to start working on it as soon as possible.

So well that’s it. Lots of work ahead of us. I think it’s going to be fun

Separation of duties – the lines are so murky – Follow up and conclusion

Taras — Wed, 16 Jul 2008 03:46:41 +0000

So after an extended investigation into the separation of duties we have determined that what we were actually looking for was a lead developer.

I am happy to report that Vitalik, a developer who I’ve been working with for a year and a half, has taken my responsibilities as a lead developer.

Basically, here is what we found.

Lead Developer is a person who keeps the project together and makes sure that everything is done in a complete deliverable form. It’s the person who will fix all of the little issues that will come up when a project is going through development. Also, a great Lead Developer know when to delegate elements of a project to other people.

In the past, I’ve found that I am often the road block in the progress of projects. This is because projects require my input to move them forward. This I think is a sign that I was a Lead Developer, who needed to delegate more.

I am very relieved that this is actually happening now. Nothing is perfect but we’re definitely making steps in the right direction.

Separation of duties – the lines are so murky

Taras — Wed, 02 Jul 2008 06:51:10 +0000

Hmm… I’ve been thinking alot about this recently and it directly impacts how we work within the cooperative.

How do all these roles separate and how do you associate tasks with roles?

For example, where does Project Architect stop, Project Manager being and Development Lead starts.

What is the responsibility that Project Manager has and how do we make it easy for Development Lead to do his work and not worry about anything extra?

What is extra for Development Lead?

hmm… so many questions.

Right now our process looks something like this.

Client Advisors works with client to determine client’s needs.
Client Advisor takes these needs to a Project Architect who is familiar with many different development tools and can advise on proper tool for the project based on price, time and resources available(this brings up a point, Project Architect does not know what resources are available because he/she is strictly technology… hmmm)

This puts a damper into my post.

The problem is that it appears that we can not seperate Project Architect from Project Manager from Development Lead, because Project Architect can suggest what technology to use based his/her oppinion of what will be best for this project.

I think I just found my answer I think the answer is the Abandance principle. We must assume that all skills are available inside of the cooperative.

Which means that any technology that Project Architect chooses for the project will be implementable by the Beecoop members.

What does this mean for the process?

This means that process flows in the following way:

Client Advisor works with client to determine needs
Client Advisor takes needs to Project Architect
Project Architect recommends a technology to use for the particular project, including the CMS, modules and other elements that might be necessary
Project Architect creates the project, indicates which skills are necessary for this project and either requests particular memebers to participate in this project or publicizes to Beecoop that these skills are necessary. People will fill in the roles and indicate the time necessary to fulfil their element of the project. (development lead is also selected based on the skills necessary for this project)
Project Architect assembles all the information from Developers and Development Lead and provides the information to Client Advisor
Project Architect works with Client Advisor to prepare a quote that includes, cost, time and technology
Client Advisor goes to client and presents the information
Client Advisor gets deposit and pays Project Architect for his/her role in building project requirements and costs
Client Advisor pays deposits to all developers and development lead and the development process begins

Not too shabby only 9 steps.

ok, this is a mind dump, so it’s not perfect. I’m looking forward to your feedback.

Credits for my new blog.

Taras — Wed, 18 Jun 2008 16:06:20 +0000

I would like to dedicate my first blog post to the people who made it possible.

Designer: Elysha Poirier checkout her company site to see her portfolio, she’s greAT!
Template: Vasile Rusnac – good job man Spacibo.

And last but not least, me

Thanks for reading the first post on my blog.

Taras