BBWT Technical Platform

Session Expiry

Load a website, login, then go have tea – what happens?

In BBWT, after a configurable time-out, the system will show a banner message at the top of the screen: “You will be logged out in 30s”. This counts down, and at the end of the time the system will redirect to the login page.

Multi-Tab Handling

Some users open multiple tabs in their browser and try to login to the same site twice. On many sites, this causes serious problems, as the site only accepts one login.

BBWT handles this properly: when we get a second login (whether a new user or same user) then all tabs switch to be the latest user session.

Caching + Compression

Web applications typically send a lot of files and images to the browser on first load. Well-designed applications make good use of file compression and the browser cache, so that on the second load of the site, almost nothing is read from the server, saving time.

But this often causes a serious issue during development, where the tester’s browser doesn’t see the new version immediately, leading to confusion.

BBWT uses a popular tool, WebPack, which completely solves this problem: files and images are compressed and uniquely identified, so that caching works effectively and correctly.

Look and Feel

BBWT uses the PrimeNG library to provide the core GUI controls – menus, data input, grids, etc.

Customers are free to choose skins from the PrimeNG site which change the visual appearance of the site – some of these require the payment of a small fee. It is also easy to change the key colours used.

But BBWT is just a starting point: all applications are still built to order to custom specifications, so with BBWT you can make any changes required to the style of the application.

Design Guidelines

PrimeNG provides a good set of GUI building blocks but creating a polished and consistent application still requires care and attention. Blueberry has created a set of design guidelines which help our teams to design screens for customers in a consistent way.

Grid Options

In modern business applications, the display of data in a table or grid is a very common requirement. Blueberry has worked to customize the PrimeNG grid to:

• Provide multiple ways to support master-detail editing of records within the grid.
• Support advanced filtering of grid records.
• Allow for inline, popup and new-page styles of record editing.

Delivering a powerful and polished grid UI goes a long way to making applications more power-user friendly!

BBWT delivers modern web applications which can work on all modern browsers including mobile browsers. It also supports Progressive Web App (PWA) development, providing options such as:

• App auto-update prompt.
• App installation prompt.
• App offline indicator.

The BBWT GUI is responsive: web pages and tables adapt to the screen size, so they can be viewed on mobiles and tablets. But this is not the whole story:

• Pages that feature very wide content (e.g. grids with many columns) simply won’t work on Mobile – they need to be redesigned or excluded from the mobile pages.
• Main system pages which have scroll bars or complex design may not be easy to use on tablet, so again consideration needs to be made.
• Full support for mobile / tablet often requires gesture / touch support, which needs to be handled separately.

For customers with a very strong mobile requirement, we can build an app which interfaces to the system over web services.

The goal of BBWT is to simplify the development of powerful web applications by delivering pre-built building blocks. To achieve this, we’ve included the following application features:

XLS / CSV Import

Implementing a polished and reliable CSV import can be challenging – there can be instances where you might need to handle files with millions of rows, so the need for reliable and consistent error handling is crucial.

BBWT includes a robust CSV Import module which meets these goals.

Document / Photo Management

Many applications need to store documents and/or photos. BBWT includes a document / photo management module which stores files securely within Amazon S3 (it can also be switched to Azure Object Storage). The module supports upload of new files, display in the application GUI of file details, and download.

Email / Email Templates

Most applications need to send emails at some point – even if it’s just to send a password reset link. With modern email, we need to use a password-protected account to send email. BBWT settings include this option.

In addition, BBWT includes an Email Template editor which allows the administrator to edit the text and formatting used when sending an email. This allows some branding to be introduced, for example.

BBWT has been designed to meet the Open Web Application Security Project (OWASP) protocols and has a full suite of modern security features – including secure user authentication (including 2FA options) and a comprehensive user permission management system. And as you’d expect, security starts with the login process.

Secure Login

1. The login process includes an on-screen indicator of password strength.
2. Passwords are not stored in the database. We use the ASP.NET Core User.
3. Management features to store strong hashes of passwords.
4. Passwords are never known to anyone except the user.
5. Password Reset requires that an email be sent to the user with a time-expiring link. The user uses this link to reset their password.
6. If a user changes their email address, an email is sent to the previous address to confirm the change and noted in the UI that the email is not verified until the link is clicked.
7. When a user creates a new password, it can be optionally checked in the online database of insecure passwords – haveibeenpwned.com.

Admin Login

• Administrator Pages are held on a separate system section: non-admin users do not download any admin-only pages.
• Access to the Administrator Pages (technically the web-services) can be secured by an IP white-list. Attempts to access the server-side internal API from a non-listed IP will return a permission error (this also allows customers to restrict Admin access to users on the company LAN or accessing via VPN).
• Admin users can be set to require 2FA authentication.

Two-Factor Login

• BBWT supports Two-Factor login using the TOTP standard, supported by Google authenticator and many other authentication Apps / tools. We have recently added support for the FIDO U2F standard for physical USB security keys – the user must insert these into the PC to login.
• Users can turn on Two-Factor, and the system will display a shared secret / QR code which can be scanned by Authenticator Apps to setup the connection.

SAML

• BBWT supports SSO via SAML – this allows us to use external Identity Providers from e.g. Google or Microsoft to authenticate users.

User Management

BBWT supports comprehensive user management, with support for Users, Groups and Roles/Permissions.

In practice, the detail of exactly how users are managed depends on the application:

• Some applications have a flat user structure – there’s one set of users, perhaps organised into Groups.
• Other applications are designed to allow end customers to login and manage their own organisation within the system.
• In the most complex case, there is a multi-level hierarchical structure of (for example) sales regions or customer offices/sites.

The core functionality of BBWT supports Users, Groups and Companies, and further customising is possible to support exact customer requirements.

We have also implemented user Impersonation, which allows a support user (with the right permissions) to temporarily impersonate an end-user, so they can see exactly what the end user sees.

Application Security

BBWT uses a “Single Page Application” approach: the web-pages and client-side code are all downloaded on first visit to the website. The rendering of these pages happens client side. The server provides a web-service API to deliver the data required by the client.

Security within BBWT is handled using the ASP.NET Core model of permissions and claims. All API methods are marked with the roles that are permitted to access that API method.

Within the Administrative Pages, we have a page which shows a list of all API endpoints, and the corresponding roles that are permitted to access that endpoint. This is generated automatically from code meta-data and is extremely useful in allowing project and testing staff to see the rules that are in place.

Security Testing

Blueberry internally uses a popular web-security tool to scan both the core BBWT system and customer applications for vulnerabilities on a regular basis.

In addition, a number of Blueberry customers have commissioned external penetration tests – these have identified issues, which have been fully resolved within BBWT Core.

Custom development is a complex and expensive business, which is why we created BBWT – to try to reduce the cost of development through automation and thereby streamline the development process.

Real-Time Editor

Developing a custom web application is a complex process, which is normally broken down into several development stages. As the application moves from one development phase to the next, each phase will invariably require changes or fixes to improve the completed parts of the system, usually via feedback requests received by the development team.

Considering that a change request may involve multiple people – the Customer, Project Manager (PM), Tester and Developer – the process can be time-consuming and increase the cost of the project. It therefore makes sense to try to minimise the cost of development by automating this process as much as possible.

BBWT’s Real-Time Editor (RTE) feature provides a way for just one person to directly edit the source code mark-ups (HTML) in an application’s pages, so removing the need for feedback requests to go through many hands before being implemented. The RTE feature enables edits or new elements to be applied directly to test pages, which automatically updates the source code in real-time too.

The editor is activated by a pen icon in the top menu and remains active until the user closes it from the menu or logs off. Page’s reloading doesn’t cause the edits made to be lost or change the editor’s on/off state.

The highlighted pen button in the screenshot below demonstrates the edits counter. The counter indicator is a ratio of the number of edits made during the current editing session (“2” in the screenshot) to the number of total edits being applied to the GitLab repository’s code (“5” in the screenshot).

The menu items available in the RTE are:

• Submit to PM: this sends edits made during the current editing session to GitLab as a commit, then the CI/CD job is immediately triggered, processes the edits and applies them to the repository’s code. After submitting, the editor remains active.
• Close Editor: this closes the editor without losing the edits made during the current editing session.
• Cancel New Changes: this cancels the edits made during the current editing session, stored locally in the browser. It’s possible to undo them with this menu item.

For cases where fixes are related to static page content, RTE has the potential for delivering huge savings in time.

For example, the UI part of the real-time editor provides these options:

• Adding a tooltip message to an element (input, label etc).
• Adding/changing a phrase of a container element (like <p>, <div> etc).
• Fixing labels. For example, title/sentence cases’ fixes.
• Fixing control’s attributes (like input’s placeholder etc).
• Fixing a URL of a link element (<a>).

Based on this approach of directly editing mark-ups within the running application, the RTE feature has the potential for even further improvement in development automation.

For example:

• Integration with localisation.
• Automatic validation/correction of page content by the RTE script (for example, the script can automatically correct title/sentence cases for all headers/labels/buttons).
• Adding comments to the mark-up with instructions for a developer (for example, the PM/customer adds a comment near the submit button of a form with a request to show a special ‘toast message’ on saving).
• Changing mark-up element’s styling.
• Adding custom elements to the mark-up (for example, the PM can add a button to a form and then the developer implements the button’s behaviour).

Why is this useful? Page content generally needs to be specified by the customer, because it is the customer’s system, and they have the domain knowledge. In previous systems, customers needed to write a long list of fixes to send to us to include in the next iteration. This was a time-consuming process. By automating this process, we save a lot of time and make it easier to ensure that we have correct content in place.

Database Explorer

At the heart of most modern systems is an SQL Database – which to the non-developer is effectively a collection of Excel tables, holding all the data the system works with. The structure of the database is known as the schema – this is the list of tables and fields, and the relationships between them.

Knowledge of the database schema used by a complex application can be very useful, particularly when a customer has technical staff who might be working directly with data exports or reports themselves. But documenting the DB schema is often a challenge – the structure evolves rapidly during development, and the tools that exist for DB documentation are complex.

To address this problem, Blueberry has built into BBWT a database schema exploration / documentation tool. This feature is obviously only accessible to administrators (and can be removed from the system if not required).

This tool allows Blueberry staff to add notes / comments to all tables and fields, and keep this documentation even as the fields and tables change over time. It also provides a simplified view of the DB structure – hiding unnecessary tables – which make it easier for customer technical staff to understand the system.

But why reinvent the wheel? By integrating this functionality into BBWT, it is always to hand, and immediately available. No need to install SQL tools, nothing to learn. In our view, this makes all the difference.

Data Anonymization

With GDPR, there is often a requirement to generate an anonymised version of a live system for testing purposes. Blueberry makes use of a tool called Anonimatron to assist with this – see anonimatron.

We are working to integrate this tool into BBWT. The database explorer module includes support for labelling fields with a “type” – e.g. email, DOB, Name. This type is used to specify the anonymization rules that should apply to the field – the system then generates an XML which Anonimatron can use to make an anonymised copy of the live system database. In future, we’ll have the option to do this as part of our build systems, so every time we release a new version to live, we can automatically copy an anonymised backup to the test system.

Data Dictionary

This feature allows the database schema to be marked-up with additional meta-data, such as types, typical field widths and validation rules. This meta-data information is then made available to the GUI code where it can be used to automatically deliver client-side validation.

For example, perhaps a system has a “Person” record with an “email” field. In the DB Browser, we set the type of this field to email, and in the GUI code we link the GUI control to the correct field. The GUI code then automatically applies Angular validation rules suitable for an email field (these rules are defined interactively within the DB Browser – there’s an admin page to set the rules for anything of type ‘email’).

This mechanism allows core developers to focus on the hard parts of the system – the effort of getting the validation rules right for each field can be left to testers and technical project managers – and customer staff.

Of course, the built-in logic can be overridden where specific application requirements need more specialised behaviour.

Reporting is a core part of many applications, but it is a complex area: there are many popular tools for making reports, and customers often have their own preferences. Also, in recent years a new collection of BI (Business Intelligence) Tools have arrived which customers are interested in.

For BBWT we've developed a powerful Report Builder, which generates reports from a database in run-time for end-users. BBWT also has a database documenting feature, called Database Explorer, which can connect to external DBs to BBWT and be queried.

Coded Reports

We built reports directly into the application as web pages. Although this requires developer time, it has some significant benefits:

• Reports are completely integrated – there is no jump out to another system
• The code is completely custom, so we can do anything required
• It uses the normal authentication and security mechanisms
• There are no additional licence costs – no reporting server costs or licence costs
• Reports can be made available to any system user

The downside is that we have to spend developer time on the reports, and they cannot be created by customer users.

In general, we find that this approach – although non-customisable – is popular with customers, because it eliminates the need to think about a separate infrastructure for reports.

SQL Server Reporting Services (SSRS)

SSRS is Microsoft’s main reporting tool for SQL Server, and it is a powerful and sophisticated tool which has powered reports on many systems for years.

Microsoft are now promoting their new BI system – Power BI – very heavily, and it appears they are suggesting it as a replacement for SSRS. The truth here is a little elusive, as in practice the two systems arguably have different purposes and can be integrated.

SSRS has a few limitations:

• It is licenced as part of MS SQL Server. In systems which have a light or occasional reporting requirement, this is fine. If the reports become large or complex, it becomes necessary to deploy a second SQL server to deliver the reports which requires a new SQL Server licence.
• It is typically not supported on cloud-hosted SQL server installations, such as AWS RDS. This makes cloud deployment more expensive.
• It is an external system, and it can be difficult to get the styles exactly the same as the main application. Most customers don’t mind as long as the reports look reasonable and have the same key colours and logos.
• A valuable feature of SSRS is that it can generate reports in PDF, Word and Excel automatically – however, making all formats look exactly the same, and eliminating formatting issues can take a lot of extra work, so this feature is not as free as it seems.

Power BI

The following information is correct as at November 2018.

Power BI is Microsoft’s new BI / reporting tool – it’s a complex and powerful product, with many features. Microsoft is putting a lot of marketing effort, and it is being seen as good tool by customers.

As with many Microsoft products, the true story behind the marketing is complex, and pricing is quite opaque and complex. The following is Blueberry’s current understanding:

• Power BI is first and foremost a BI tool, not a reporting tool.
• BI tools traditionally take a copy of source data which is added to a Data Warehouse – a specialised database – where users can then analyse the data. Reporting tools usually work against live data. Power BI can do live reporting, but the toolset in general assumes the user is importing data, or scheduling extracts from another system. Making Power BI work against live data is not impossible, just requires a bit more effort. Of course, some customer requirements may work perfectly well against a copy of the data from the previous night – but many reports want to have live data.
• Power BI seems to make it harder to put together a traditional looking report – i.e. a screen with a set of filter controls at the top where a user could select the year to run the report for. It does allow input controls to be added to the report, but this seems to be slightly discouraged in the user interface. The intended user of Power BI is assumed to be a person who is doing the filtering, and then writing out a final report – but this makes the system less suited to general reporting.

On the cost side, we understand:

• The desktop version is completely free, but doesn’t allow reports to be published.
• Reports can be published free as globally open reports on Microsoft servers – but only if they are accessible to the whole world, which generally isn’t very useful in our context.
• Reports can be published for use within an application, but it seems that all users who want to access the report need a $10 per month Microsoft licence.
• An organisation can purchase an Azure hosted reports server, which allows reports to be viewed by anyone in the organisation (and external partners) but this has a cost of $5k per month.
• An organisation building a custom web-application can use Power BI embedded. This requires an Azure server – the cost depends on the number of reports displayed per hour. The smallest server starts at £548 per month and allows 300 report pages to be drawn per hour. However, these servers can potentially be turned off when not used, and are billed per minute use.
• All of these options require setup / configuration within Azure cloud, which has some complexity. Using non-embedded Power BI from within a custom application will require some level of integration with Azure Active Directory which appears a little complex.

The bottom line is that delivering Power BI reports within a typical web application to an audience of client and their customer users will cost at least £300 per month.

We suspect that PowerBI will become popular with customers – it does have good data processing and formatting features. However, this is not an easy route to delivering reports.

Google Data Studio

Data Studio is Google’s version of an online reporting / BI tool. It runs within the Google Cloud, and as at November 2018, the core reporting tool system is completely free – although you would pay for any data / databases stored on Google Servers.

From limited testing, and feedback from one customer, Google Data Studio seems to be a lot easier to use than Power BI – users have created basic reports in a matter of minutes. However, when drilling into the detail we found more limitations:

• Filter control presentation etc. is not very pretty.
• It was oddly slow at times.
• Its reports on mobile display aren’t very easy to use – controls too tiny.
• In the better MS Report designer tools we could do arbitrary customisations of the output, e.g. “Round to the nearest thousand, present in red if negative”. We could not see how to do this in GDS. Though, the built-in defaults, and the customisation options are better than expected.
• It has limited composability: no defining and re-using presentation of a sub-report elsewhere.
• It’s not very “data intelligent”:
  
  • Data sources are individual tables in GDS, at MS data sources are typically whole databases. To link GDS tables, you have to use their UI to create a new data source of blended data, clicking on the key fields.
  • In MS Report Builder, we have the option to write arbitrary T-SQL or use a query builder. The query builder in GDS is more limited.
  • Power BI is intelligent in that it looks through the entity relationships in the database and the user just picked fields – it did the hard work of inferring what you meant.
• It’s not clear how GDS would support live / test environments.
• It’s not clear how GDS reports might be copied into source control system.

In addition, GDS does not support Microsoft SQL Server – it supports many other Databases, but not MS. There are solutions which involve synchronising the tables of interest from MS SQL into another DB, but this does seem a huge effort to go to.

AWS QuickSight

QuickSight is Amazon’s cheaper, no-frills answer to Power BI. It does not support as many visualisation types, but is cheaper and easier to get to grips with. We have not fully assessed this tool.

Stimulsoft

This is a conventional business reporting tool aimed specifically at developers of bespoke systems. It is licenced per developer at $699 – a reasonable cost. There are many similar tools – e.g. ActiveReports. From our initial research, Stimulsoft offers a good price/performance ratio – ActiveReports is $13k for a full server deployment.

The benefit of using this type of tool is that it would be fully integrated into BBWT3.

We are planning to do a larger comparative survey of these tools in the near future.

BBWT is a toolkit for making web-applications. At present, BBWT3 is designed around a traditional web-server / database combination – although we’re looking at the latest Serverless approaches, delivering BBWT3 Serverless is some months in the future.

A BBWT3 based application needs a modern Windows or Linux system to act as the web-server, and a MySQL or MS SQL compatible database back-end.

Blueberry now almost exclusively deploys systems on Cloud. We have worked with both Azure and AWS – but generally prefer AWS.

Windows vs Linux for Servers

Blueberry has primarily used Windows in the past. In modern development, particularly in California, Linux is increasingly becoming the server standard, and Microsoft have now largely stopped fighting this, and embraced Linux. Having said this, Microsoft is still extremely strong in business, and many of our business customers still prefer to host on Windows.

ASP.NET Core which BBWT is built on works with no problems on both Linux and Windows – and we are now preferring Linux to host BBWT systems. However, it is the case that there are still more tools for ASP.NET Core on Windows than on Linux, so in some cases we do recommend Windows.

We use Debian-based distributions to host BBWT.

Scalability – Load Balanced Deployment

The BBWT architecture is designed to be compatible with Load Balanced deployment – by default we store no application state or data on the web-server at all, using RESTful services.

On AWS, we’ve used Elastic Container Service (ECS) to deploy a fully load-balanced BBWT implementation using BBWT web-servers within Docker containers. We also have the option to scale AWS installation instances up/down dynamically if needed.

Important: the server-side requirements for a web-application depend totally on the peak concurrent users. If something in the business model means that everyone logs on at 9am, you will need a much bigger server than for an evenly distributed system. Predicting the server size required is difficult – it is much easier to deploy the system, measure the load generated by N users and predict from there.

Note: The advent of Cloud Computing has made it vastly easier to upgrade systems with very little effort. If a customer needs the system to support twice as many users – we can typically accommodate this in minutes.

Database

BBWT uses LINQ and Entity Framework Core, which supports multiple back-end databases. Our preferred systems are MS SQL and MySQL (we strongly considered PostgreSQL, which is excellent, but the popularity of MySQL and level of AWS support for MySQL convinced us. But if a customer strongly preferred Postgres, this is possible).

Switching between MySQL and MS SQL Server is not completely free – even though we use Entity Frameworks which hides the underlying DB, we often have to rewrite EF queries, and we usually end up with some native SQL code, which needs to be migrated when switching DB.

On AWS, we now use their Aurora DB because it’s much cheaper than MS SQL and very fast.