Over a six month period, Code Enigma assisted Oxford City Council to launch a new Intranet built in Drupal. After its launch we interviewed the project manager at Oxford, Neil Lawrence, to find out how things had gone.
Mentoring, not building
Before getting into his experience it's worth setting the context because this project is not a standard agency site build. Oxford issued an RFP for a Drupal-based intranet following a report produced by Socitm that recommended a Drupal-Sharepoint solution. We thought the proposed budget was unrealistic for a full intranet built by an agency, so we suggested that we mentor the in-house team of two developers so that they could build the intranet with our guidance. This involved us providing initial training and then putting a developer onsite part-time to act as team leader. Oxford provided two developers and Neil as project manager.
Back to the 'firsts' as listed by Neil. It was the first time Oxford had worked with Open Source, the first time they had built a web application 'in-house', the first time they had used Linux servers, and the first time they had carried out user requirements gathering with an Agile approach.
Neil carried out the user requirements gathering with a group of champions representing various sections of the Council and from that produced a backlog of desired features. Many of these related to addressing flaws in the existing, outmoded system, such as finding content and how it was organised. He also invited Oxfam, also based in Oxford, to demonstrate their Drupal intranet, to stimulate thinking about possible features. During the project, the team periodically demonstrated progress to the champions including one early session where he said their work was greeted with “stony silence” because of the absence of bells and whistles, meaning lack of visual impact. They quickly addressed that by creating some new homepage features such as a slideshow. Designers and UX experts might be sniffy about this but the reality is that an important element in any Intranet project is selling it to the users.
The Drupal effect
Even when I'm working on something else unrelated, I can immediately see how this could be done better on Intranet
The guiding mentality is now, 'You own it, you manage it'
The key features delivered by the new system are an effective staff directory, closely integrated with Active Directory and the backend personnel system, and a decentralised content authoring and workflow system. These two features address the biggest problems with the old system – finding the right person, and content going out of date because of editorial bottlenecks. Drupal enabled the team to build exactly the staff directory they need to reflect how teams are organised in Oxford. On editorial workflow, they took the radical approach of letting anyone edit content. When it does get edited, the content owner gets notified with a 'diff' of what got changed.
This model was informed by advice from Oxfam where a small team administer an intranet with over 900 separate group areas so can't possibly monitor all content. It's arguable that it's also the result of Open Source thinking in projects like Wikipedia. The guiding mentality is now “You own it, you manage it”. Would a local authority ten years ago have been willing to let anyone edit content? The other interesting thing about the Oxford workflow is that they designed it for themselves. They took a look at the Workbench module, which is often an out-the-box solution for workflow, but it wasn't right for them so they made their own. That's a great example of the Drupal advantage over a conventional product; if it doesn't do what you want you can customise and you don't have to be an advanced programmer to do it.
Other features include the media module with some custom views to enable searching for images by tags, a simple document management system, and web forms. Constraints on time and budget meant the Sharepoint integration that would have provided fuller document management has been postponed and there are still a lot of forms that are actually just downloads for completion but as Neil says “at least they are easy to find”. That ease has been achieved with Apache Solr search addressing a common complaint with legacy systems, that stuff is hard to find and you have to drill down through menus. Just use Solr and Drupal taxonomy so you can get smart 'google-style' text search plus faceted search by tag. The team also focused on making improvements to the edit interface to make editing as painless as possible. Again that's addressing the business problem with intranets that they go out of date because they can be a pain to use.
The mentoring approach
So, Oxford now have a working intranet they built for themselves in Drupal. It doesn't have all the features they want. In fact, Neil thinks the original minimum viable product was unrealistic, but there is a working application and a road map for new features such as email newsletter and motions to Council. More important, the Council team can actually plan and implement the road map for themselves. “We know how to go about solving the problems”.
Problem-solving is the essence of being a Drupal developer so it's reassuring that the Oxford team feel confident about this, as a result of the mentoring model. Neil explained that rather than provide a solution, our developer James Panton would outline the options. “James can just turn left immediately. He doesn't just do Agile, he thinks it”. Obviously that's good to hear about one of our team, but we think there's also a learning point here about the attitude that's needed to make the most of Drupal's potential. If he did have a criticism, Neil felt we could have driven the internal team a bit harder, and it may be a risk of a mentoring model using Agile that the team lose sight of the need for binding milestones.
Staying on process, Neil really enjoyed using Agile for the first time. In particular, he got the importance of a project having a product owner and he was up for doing it while using the champions group for capturing stakeholder wishes. In fact, he confesses to finding it hard to let go – the intranet feels like one of his children. “Even when I'm working on something else unrelated, I can immediately see how this could be done better on Intranet”.
He also enjoyed the structured way we apply Agile using standardised documentation, scrum diaries, and detailed task breakdowns for each feature or story. Although we tend to do task breakdowns to facilitate estimating even on internal projects, it's particularly important when mentoring an internal team so that they have a record to refer back to for future projects.
So far the story has been very positive but there were also some major hurdles to overcome. Looking back Neil acknowledged that the budget was unrealistic and had been set before any requirements gathering had been done. There should also have been more up-front consultancy to investigate challenges like data migration. That certainly chimes with our experience on other projects – cutting down on initial consultancy and discovery work is always a false economy.
Although mentoring the Oxford team was a good pragmatic approach it wasn't a panacea. The learning curve proved steep with the team having to learn tools like Bash and Git so they could cope with a professional continuous integration setup.
The team didn't factor in the need to have a Linux server within a walled garden hosting environment that can't be accessed from outside, making it impossible for Code Enigma staff to assist remotely.
There was also a false start caused by using the Drupal commons distribution, on the false assumption that this might cut down on design work. Stakeholders found it to be “too Facebook” and stripping out unwanted features took longer than building from scratch. This feels like a common problem with Drupal distributions – if they do exactly what you want they are okay but if not, they are more trouble than they're worth. To be fair, there are nice ideas in Commons such as the activity feed that the Oxford team copied, so a useful function of a distribution can be that it stimulates ideas.
Overcoming these problems delayed the project so it launched a few months late. However, the view within the Council was that the delay was acceptable because it was transparent why it was happening, and they essentially owned the timetable.
James can just turn left immediately. He doesn't just do Agile, he thinks it.
The project sponsor's view
In addition to reviewing the project with Neil, we discussed it with Jane Lubbock, who acted as project sponsor. She felt the project had provided a great opportunity to learn about working with Open Source and what's possible with a powerful framework like Drupal. Her preconception was that Open Source software would be fairly basic so the sophistication of what can be built in Drupal was a surprise. On the other hand, the amount of time and effort needed to build a custom solution was also a surprise, and raises concerns about the skills needed, and how the Council can ensure staff can keep their skills up to date. We discussed the possibility of staff engaging with the Drupal community and it was good to hear a manager acknowledge why that might be a useful thing for staff to do. Her view of the project process was that working with an external agency was useful to get access to design skills and knowledge of the CMS market that it would be difficult to achieve by doing everything in-house. The mentoring model was not new in that Councils are familiar with knowledge-sharing but not necessarily in the software development area. Looking to the feature we discussed how the impact of the project would be assessed. There are no quantitative measures in place but Jane felt the Council would be able to assess softer measures such as reduced frustration with some everyday tasks and improved job satisfaction from using up to date tools.
I thought Open Source software would be much more basic than this.
The developer perspective
Looking back on the project a few months after completion, James said it had been a great reminder of the value of Drupal as a tool that enables committed site builders to produce sophisticated websites without knowing much about coding. Professional Drupal developers can get so buried in Drupal's framework tools that they lose sight of what's possible with a few modules and a problem-solving mentality. One simple example is that they based a chunk of site organisation on the core Book module, which developers tend to overlook, but it made perfect sense to the Oxford team.
“To be honest, I can't tell you how most of it works. I showed them Drupal and they built their own staff directory and most of the other features. I spent my time grappling with specific technical problems like data migration”.
His biggest challenges came from working with a very tightly controlled network. “Have you tried teaching Git with pen and paper?” We are used to using online tools to do things like load-testing and debugging problems with automated jobs like data imports becomes more or less impossible with no remote access. This feels like a policy area that needs reviewing. How are public bodies going to comply with a 'digital by default' agenda if they have outmoded hosting models?
Where James did admit to adding value was in teaching professional development techniques and tools such as version control, Bash, and SCP, plus Drupal-specific tools like Drush, Ctools and Panels Everywhere. It was tricky for the team to get hold of the difference between content and configuration, and the need to get configurations into code wherever possible. Many of these techniques are essential for working as a team, but it's easy for teams that are used to them to forget that much web development gets done by people functioning as individuals rather than an integrated team. It's also easy for us to take basic knowledge for granted, such as how to select suitable modules using information from drupal.org.
The Code Enigma view
The smart move would have been to avoid this project when it first appeared; the budget was too small and the expectations were unrealistic. We didn't do that because we had already started mentoring some other organisations so we thought there was a possibility of achieving something on a tight budget. There was still a big risk that guardians of public money might like the sound of a collaborative process but still insist on delivery of an impossible specification. The fact that didn't happen is a tribute to a brave project manager backed by imaginative managers. They completely got Agile principles that they own the project and the development team can only deliver what's possible in the time available. The result may not be the World's most advanced intranet application but it is an application that was built by Oxford, and they know how to extend it and take it forward; that feels like a vindication of how we work and the value of Drupal.
To be honest, I can't tell you how most of it works. I showed them Drupal and they built their own staff directory and most of the other features. I spent my time grappling with specific technical problems like data migration.