BI Architecture
Part of Selecting BI and dashboard software
Evaluating BI tools using the same dataset
Build a fair BI software trial with one data snapshot, approved figures, matching user tasks and recorded configuration differences.
Compare BI tools on one controlled business case: same source snapshot, measure definitions, periods, user tasks and expected figures. This shows workflow differences without identical internal designs. Record the preparation and configuration each product needs.
Prepare a reference case
Choose data you are authorised to use in a trial. Include a transaction table, a useful grouping field, more than one period, and cases that reveal errors — for example a cancelled record or an order with several lines. Remove personal detail the evaluation does not need.
Document what one row represents, which date assigns it to a period, what the headline measure counts and what it excludes. Prepare reference results for a closed period: an overall figure and a few breakdowns. Have the measure owner approve them before you assess candidates.
Keep the input stable. If one product uses an extract and another a database connection, preserve the source snapshot and record the connection difference. Do not silently correct a data issue for only one candidate.
Give every candidate the same tasks
Ask an intended author to import or connect to the data, reproduce the approved figure, build a period trend, and investigate an exception through a breakdown. Ask an intended viewer to find the result, change a filter, and explain which records are included. Have the model owner request one controlled change to a common measure.
Record for each task / Why it matters
- Expected and obtained figure, with filters
- Separates calculation errors from presentation differences.
- Source snapshot and update point
- Prevents a newer extract being mistaken for a better tool.
- User role and effective permissions
- Shows whether the intended person can complete the task.
- Preparation and workarounds
- Makes implementation effort visible.
- Time and assistance observed
- Captures the workflow beyond a demonstration.
Set acceptable results before the trial. A quick chart with the wrong population fails the accuracy check. A matching grand total can hide offsetting errors, so inspect breakdowns and the record identifiers behind discrepancies.
Keep configuration differences visible
A shared dataset is implemented differently across products. Power BI's managed self-service guidance describes authors using centrally maintained semantic models. Looker uses LookML to define fields and relationships queried in Explore. Tableau Cloud uses licences, site roles and content permissions to govern what users can do with published data and views.
Test each product through a suitable configured workflow, then record which work needs a specialist.
Before treating a failed task as a missing capability, check the configured role, licence and settings. Do not mark a task as passed merely because an administrator can do it when the intended author cannot.
Interpret the result narrowly
Review accuracy first, then author effort, reader usability and operating effort. When observing responsiveness, record data volume, connection mode, configuration and cache or refresh state. A small trial cannot establish performance under the full production workload, concurrent use or future growth.
The result is a decision record, not a universal score. State which candidate met each requirement in the tested configuration, which discrepancy belongs to the data or definition, and what remains unverified. Keep the snapshot and reference figures for checking the chosen implementation after deployment.
