Splunk Enterprise Software

What is Splunk?

Splunk Enterprise Software (“Splunk”) is a powerful tool for searching and exploring data.

Splunk is often used by system administrators, network administrators, and security gurus, but its use is not restricted to these audiences. There is a great deal of business value hidden away in corporate data that Splunk can liberate.

Who uses Splunk?

Splunk is a powerful platform for analyzing machine data, data that machines emit in great volumes but which is seldom used effectively. Machine data is already important in the world of technology and is becoming increasingly important in the world of business.

It’s used by:

  • Security offices
  • Marketing departments
  • System administrators
  • Network administrators
  • Application development teams
  • Application support teams

splunk_usage

How does it work?

  • Splunk begins with indexing, which means gathering all the data from diverse locations and combining it into centralized indexes.
  • Using the indexes, Splunk can quickly search the logs from all servers and hone in on when the problem occurred.
  • Splunk can then drill down into the time period when the problem first occurred to determine its root cause. Alerts can then be created to head the issue off in the future.

Splunk provides one repository, data indexing, search & visualization for your data.

What are the potential Splunk data sources?

During indexing, Splunk can read machine data from any number of sources. The most common input sources are:

  • files: Splunk can monitor specific files or directories. If data is added to a file or a new file is added to a monitored directory, Splunk reads that data.
  • the network: Splunk can listen on TCP or UDP ports, reading any data sent.
  • scripted inputs: Splunk can read the machine data output by programs

or scripts, such as a Unix® command or a custom script that monitors sensors.
Technically speaking, retrieved events from your indexes are called “events.” If those events are transformed or summarized so that there is no longer a one-tone mapping with events on disk, they are properly called “results.”

Announcements at the Splunk 2015 Conference in a slide

splunk_news

SPL (Search Processing Language)

The Search Processing Language encompasses all the search commands and their functions, arguments and clauses. Search commands tell Splunk Enterprise what to do to the events you retrieved from the indexes. For example, you need to use a command to filter unwanted information, extract more information, evaluate new fields, calculate statistics, reorder your results, or create a chart.

Some search commands have functions and arguments associated with them. Use these functions and their arguments to specify how the commands act on your results and/or which fields they act upon. For example, use functions to format the data in a chart, describe what kind of statistics to calculate, and specify what fields to evaluate. Some commands also use clauses to specify how to group your search results.

There are four broad categorizations for all the search commands: distributable streaming, stateful streaming, transforming, generating.

splunk_spl

Where can I find more information?

http://docs.splunk.com
http://docs.splunk.com/Documentation/Splunk
http://splunkbase.com
http://docs.splunk.com/images/a/a3/Splunk_4.x_cheatsheet.pdf
http://answers.splunk.com/
Reference Guide http://www.splunk.com/web_assets/pdfs/secure/Splunk_Quick_Reference_Guide.pdf
Educational Videos http://www.splunk.com/view/education-videos/SP-CAAAGB6
Splunk Blogs http://blogs.splunk.com/
Splunk Wiki http://wiki.splunk.com/Special:SplunkSearch/wiki?q=your-query
Certification and education http://www.splunk.com/view/education/SP-CAAAAH9

Print Friendly

Elastic Search

Elastic Search is a document oriented search engine based on JSON & Apache Lucene (java library). If I have to explain in plain language Elastic Search, I would tell you it’s a search engine, algorithm, technology to search big amounts of data(documents) in a very optimized way. Of course the easiest is to tell you it’s a Solr like technology with a bit more powerful search and aggregation capabilities. It can be invoked from a web application or by a process. You can define search parameters or indexing.

 In Elastic Search you can do structured search (filters), full-text search (query), and analytics (aggregations).  

When using analytics (aggregations) you need to add more complex aggregations, percolators and may be scripts.

It’s schemaless (schemas are created dynamically). You dont need to define in advance the structure of the data you are feeding for indexing.

There are free plugins like Kibana you can use to monitor your Elastic Search servers and performance.

Kibana

Elastic Search uses log4j for logging application errors and I would like to see that configured to log to a database as well as the local file.

How do we run queries against Elastic Search? There are multiple ways to do that. A favorite option is Postman. https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=en or Kibana Sense.

Application Support Elastic Search role skills/knowledge needed:

– Elastic Search DSL (Domain specific language) is the query language in Elastic Search (for debugging)
– JSON
– Elastic Search understanding
– Kibana understanding, for monitoring/reporting and configuring monitoring/reporting
– Knowledge of the implementation
– AWS/ Apache Lucene Java library/ Linux / JVM
– Knowledge of the server/application setup

laptop

Print Friendly

Microsoft System Center

Microsoft System Center is a suite of tools to help you manage physical and virtual servers, client computers and devices.

There are different components of System Center. In 2012 it became a single product. You can use any of the products with one license.

  1. VM Manager
  2. Service Manager
  3. Orchestrator (Azure Orchestration)
  4. Configuration Manager
  5. Data Protection Manager
  6. App Controller
  7. Endpoint Protection
  8. Operations Manager

Microsoft System Center is changing and evolving.

  • Changing potentially to a service on the Cloud
  • 1 in 5 machines in Azure currently is on Linux and Microsoft is trying to accommodate them

What Does Microsoft System Center Do?

It’s an evolution (started as managing desktops, became a configuration manager). In 2012 it became a standalone product. It’s cross platform – it works with Azure, AWS- it’s  agnostic.

  • Patching on cross platforms
  • Deployments
  • Infrastructure monitoring
  • Application performance monitoring
  • Dynamic application discovery (with Blue stripe)- it sniffs ports, with a light agent, the physical mapping of the application
  • SQL backups
  • Azure backups
  • Security and audit
  • DR
  • Automation
  • Updates client computers and devices across physical, virtual, distributed and mobile environments

What is Operations Management Suite (OMS)?

It sits on top of System Center. It can be on the Cloud or on the premise. It requires an additional license. OMS offers:

  • Log Analytics
  • You can do custom dashboards
  • Ties easily AWS & Azure
  • Nice web platform/view (HTML5)

It has 260+ business intelligence packs (plugins).

OMS

Print Friendly

AWS Abbreviations

SQS- Simple Queuing Service

SNS- Simple Notification Service

WAM- Workspace Application Manager

VPC – Virtual Private Cloud

DC (DX)- Direct Connect

EFS – Elastic Filesystem

VPE – Virtual Private Endpoint (?)

NAT – Network address translation

EMR- Amazon Elastic MapReduce (EMR) is a web service that uses Hadoop, an open-source framework, to quickly & cost-effectively process vast amounts of data.

IOPS- Input/Output Operations per Second

NFS – Network File System

AML – Amazon Machine Learning

NAS – Network Attached Storage

S3 – Simple Storage Service

EC2- Amazon Elastic Compute Cloud

RDS – Amazon Relational Database Service

ELB – Elastic Load Balancing

Aurora – Amazon’s MySQL-compatible relational database management system (RDBMS)

Print Friendly

Right Scale – Manage Your Cloud

Right Scale is a tool to help us with managing a multi-cloud environment. It’s a single interface for multi-cloud support (example AWS, Azure, Google Cloud, as well as VM).

Right Scale is SaaS (software as a service).

The features:
– Cloud and configuration management
– Credential management
– One place to manage user access to the multi-cloud environment
– Managing and automating server patching & maintenance
– Server deployments automation
– Monitoring & alerting (including backup alerts)
– Cloud cost management (cloud analytics)
– With the Collectd plugin you can collect server statistics (in numbers)
– There is the  Multi Cloud Marketplace which gives you access to server templates and scripts (ex. Chef recipes, right script and so on)
– RightScale APIs (everything that’s in the interface, can be done through API)
– It has Chef integration. It has New Relic integration.

The main focus of Right Scale:
* Automation (in and across Clouds through Self-Service and CAT files)
* Governance (Offer self-service IT access while sleeping at night. Design and enforce policies based on budgets, configurations, and user access. Control resource placement and capacity.)
* Economics (Through Cloud Analytics and Self-Service Scheduling we are able to control waste)
* Ecosystem (Multi-cloud & hybrid environment)

Note: Hybrid is the combination of private Cloud (VMware, Open Stack & Cloud Stack environment) and public Cloud (ex. AWS, Azure, Google…).

There are 3 parts to Right Scale Cloud Portfolio Management:
Self-Service, Cloud Management, and Cloud Analytics

RightScale1

Cloud Analytics – shows the cost of the cloud (User oriented view)

  • Budget Alerts
  • Spending Reports
  • Estimation Tool (Scenario Builder)

RightScale2

Want to read more? www.rightscale.com

Print Friendly

The HoloLens and Holographic Academy Experience

I consider myself extremely lucky, part of the future, special and being in the right place at the right time.

This is the HoloLens website.

As part of attending the Microsoft Build conference I got a chance to attend the Holographic Academy. I was one of the first few hundred developers to try developing an app for HoloLens. It was four and a half quality hours in a hotel hall with a big Microsoft team and magic all around. No cameras or recording devices allowed- so I don’t have photos to share. It’s still a lot of secrecy and unanswered questions but what a spectacular unveiling of the technology of the future!

Personal Setting
Somebody came with an optician tool to measure something about my eyes. I don’t understand it all but I believe it measures the space between the pupils. I got 60 as a number and I think it was the smallest number from the people around me- go figure.

The HoloLens customizations and deployments can be done wirelessly but in our with unreliable network, we had to connect the HoloLens to the computer through an USB cable. As a first step, I needed to access the device through the browser with an IP address & port number. You access a website where you plug in the number to customize your HaloLens.

The Development Tools
The tools you need are Unity for the 3D work (https://unity3d.com/) and Visual Studio 2015 for the compilation of the code and deployment.

I opened an existent application, built it in Unity and compiled/deployed it from Visual Studio. When compiling it, I had to look at the spot where I expected to see my hologram. Once the deployment finished, I disconnected the cable and I was free to move around with my hologram – a small race toy car. With a small tab in the air, I was able to move it around.

Space Recognition
HoloLens have space recognition- you see the space around so my car would fall from the edge of the coffee table and get stuck on impact with my backpack.

Building Apps
I started with a new project in Unity. Microsoft provided us with assets to use for the app. There are 3 main components/sensors you code for – gaze, tap and voice commands! We would add C# script for gaze, tab and voice recognition and attach them to objects.

Gaze
The little ring cursor will point at the object when your eyes/gaze reach the object.

Tap
The little tap with the index finger in the air will trigger a command.


Voice Recognition

I was able to implement in the code my personal commands. I was impressed how good the voice recognition was. It was easy to implement my commands for reset and drop the object. I think HoloLens actually interpret my accent better than the people around me.

The App
With the provided assets and code, my app had a small board with objects in it. There were two balls in the air that I was able to drop with a tap or a voice command. When the first ball dropped, an exposure would happen and the underground world would get exposed. I would place my gaze on the second ball and tap/give a command and the ball will drop in the underworld and I would be able to follow with my eyes the ball.

Interaction
I would be able to give my HoloLens to somebody else and they would experience my world. We were not able to interact with each other in one augmented reality. This functionality will be there in the future.

The Summary
When I saw the press release video from Microsoft in January, when I saw the demo at the keynote of //Build, my reaction was- no idea how this can be real. The experience with my hands on the HoloLens and the SDK surpassed my expectations by far! The augmented reality is very real, the HoloLens are comfortable and writing an app for HoloLens was not as hard as it sounds. I probably cannot explain the extent of my excitement and fascination. I have dreams now how to use the HoloLens, how to build apps for it. And after dreams and vision, now it’s the time to roll my sleeves and learn Unity…

Print Friendly

The Difference between Microsoft Azure and Amazon AWS

At the end of 2013 Microsoft became a visible cloud provider and Amazon was already leader. In July 2014 the two leaders in the market of public Cloud are Amazon and Microsoft. In the future Google and IBM are probably going to join the hyper scale cloud providers (Amazon and Microsoft).

AWS Free Tier vs. Azure Free Trial

An Amazon AWS free tier and a Microsoft Azure free trial are offered. It’s easy to sign up for both– you need only an email, phone number and a credit card. You will have limited instances and resources but it’s great for training or a test. You will have a basic support and access to resources like forums.

The AWS free tier is a monthly recurring program. You can use EC2, ELB, EBS, S3 for up to 12 months. It allows for Micro server (with Windows and Linux, EBS, Cloud watch, billing alerts, ect.). The charges over  the free trier will be automatically charged to the credit card.

The Azure free trial is a 30 day trial of $200 worth of services for 30 days. When you exceed $200 you will not be charged automatically, the resources will be decommissioned but they are not gone. An MSDN subscription includes $100+/month Azure services and a discount on VM’s.

Features

The naming of features is different but here is the mapping.

AWS EC2 = Azure Virtual Machines
AWS VPC = Azure Virtual Network
AWS RDS = Azure SQL Database
AWS ELB = Azure Traffic Manager (load balancer)
AWS Route 53 = Windows Azure name resolution (DNS service)
AWS EBS/S3/Glacier = Azure Storage
AWS Direct Connect = Azure ExpressRoute
AWS IAM With MFA = Azure Multi Factor Authentication
AWS Security Groups (more advanced than Azure EndPoints) = Azure EndPoints
AWS SNS/SES = Azure Service Bus
AWS EC2ConfigService = Azure VM Agent
AWS SQS (simple queuing services)/Auto Scale (more advanced than Azure) = Azure Scheduler
AWS CloudFormation/CloudWatch/AutoScale (more advanced than Azure) = Azure Automation

Both support license mobility (bring your own license)- except the OS license. Both support Puppet and Chef integration from automation and scripting perspective.

AWS RDS vs. Azure SQL Instance

Both provide hosted managed database; both simplify management, deployment and automatic snapshots/backups. Both provide multi availability zones alternatives. Both have the option to BYOL (bring our own license). AWS provides MySQL, PostgreSQL and Oracle, Azure SQL is only one.

AWS VPC Peering vs. Azure Site-to-Site

AWS allows connection between two VPC using private subnets. No transitive peering allowed. Azure security is done through ACL and Windows Firewall, uses VPN to connect.

AWS Reserved Instances vs. Azure Commitment Plans


AWS Reserved Instances (EC2)
are in three tiers (Low, Medium, and High). There is an upfront fee with a discounted rate per hour. Offer 1 to 3 year term with upfront fee. You can sell them on the Marketplace.

Azure Commitment Plans are based on a monthly pay commitment. The minimum is $500/month on Pay Monthly Plan- discount between 20-32%. It applies to all resources except Storage. It’s simple to use compare to the AWS reserved instances. They are non-refundable.

Cost Structures

You need to compare the AWS reserved instances with Azure Commitment Plans. If an instance doesn’t fit you- change it! Monitor your cost and usage.

You need to approach the Cloud- it’s not enough to be a technologist, you need to be a business analyst too. The benefit of public cloud is a known cost. You need to calculate in minimum and maximum, not at an exact cost. Both Amazon and Microsoft will provide you with calculators. AWS is slightly cheaper but it depends on your commitment plan and your MSDN subscription.

AWS will bill you per hour; Azure will bill you per minute (but cash per hour is not your main factor). You should look at discounts & features.

Monitoring and Alerting

AWS offers CloudWatch:
– free monitoring with up to 7 metrics, every 5 min
– paid 10 alerts, 1 million API requests per 1 min
– ELB have active service monitoring
– Metrics for Billing
– Notifications using SNS, SES
– Provide Phone app for basic monitoring and management

Azure Monitoring:
– Basic monitoring included (CPU, Data In/Out, Disk Read/Write Throughput every 3 min)
– Verbose monitoring pulls performance metrics from server instance every 5min, 1h and 12h

Support

Free Support
27x7x365 AWS customer service (not tech. support). Technical support – some with AWS, none with Azure.
Developer Tier Support
AWS $49 per month, Azure $29 per month but you get different thing. AWS will give you Architecture Support, Best practice guidance, Client Side Diagnostic Tools- you don’t get that from Azure.
Medium Tier Support
You get Architecture Support and IAM & API for support access with AWS but that feature is not available for Azure.
High Tier Support
It’s comparable between the two.

Security Access

MFA is an absolute must for both.  Azure lacks a bit the granularity for the users and roles.

AWS offers IAM with MFA:
– Highly detailed delegation (user, role, federation)
– Security keys for SDK/command tools/PowerShell access
– Use SSL Cert for identification thru SSH or password retrieval
– Built in support for clearing local password on boot

Azure offers Highlights – MFA:
– Admin, O365, SDK
– Federation access
– User account thru PowerShell
– OTP thru mobile app, phone call, SMS – Security reports

Compliance AWS has a lot more compliance than Azure (ex. SOC 3). AWS has a government cloud.

SLAs

It’s measured in 5 min intervals. AWS will not declare an outage unless you designed your platform properly. When the service goes down, you will get a credit for the time the service was down.

Print Friendly

Public Cloud vs. On Premise Servers

In the last 20 years we have mastered building and operating the infrastructure, now we at IT are broker of services. You are managing the service between Microsoft or Amazon and end users.

On premise we scale-up, on the public cloud scale-out. On premises we assumed a reliable infrastructure (we designed, configured, manage it this way), on the cloud we should expect infrastructure failure.  In service oriented architecture (SOA) the application is aware of the infrastructure, knows how to reconfigure or recalibrate (ex. If it requires more databases- it can make database services requests). Traditionally on the premise the application is completely unaware of the infrastructure.  In order the service to be service oriented and the application to be designed for the public cloud, it needs to be written that way -the application to manage them and understands infrastructure.

On premise is fixed cost, buying upfront. With the public cloud is usage-based cost. On the cloud there is a lot of automation of the management of the instances.

Are there really cost savings OpE vs CapEx? With on premise servers we are used to overprovision. If we learn the differences between the public cloud and the on premise- we can save money.

The flexibility and elasticity of the Cloud is very helpful- you can change hardware at a reboot.

Security and Compliance

If you don’t secure the public cloud is your fault. You are provided but you have to implement it.

Hybrid Cloud

Currently there are around 94% machines on the premise vs. 6% on the public cloud.

Having servers on premise and in the public cloud (the hybrid model) is most likely the future. The cost for the public cloud is continuously going down. The public cloud will be the default and the exception will be the on premise.

Print Friendly

Microsoft Game Jam for Kids 2

February 22, 2014

At the Microsoft Office in Playa Vista, CA
Two outstanding Microsoft evangelists- Daniel Egan and Bret Stateham.
Sponsors: Microsoft, Boon Staffing, Doris Aves, Universal Music Group.
Organizer Nia Angelina Samir.

Game Jam for KidsGame Jam for Kids

50 lucky and bright children. Most of the children are 6th graders at the Walter Reed IHP and Humanities Academy. In the middle of the photo is one remarkable teacher who cares and works hard for her students. Gratitude to Mrs. Shahine.

Game Jam for Kids

9 amazing volunteers from Universal Music Group. Doris Aves, JB Marsh, Lazslo Pinter, Lana Osnas, Brian DeRue, Igor Marchenko, Meenakshi Ganesh (MSG), Bharath Chekuri, Nia Samir. As well as other volunteers who made this possible like Svetoslav Todorov, Erik Gracia and others.

Game Jam for Kids

Big smiles from all the children.

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

We had a audio recording room.

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

The children were focused and worked hard.

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

The children had a chance to present the games and art work they created.

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

The volunteers.

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

Game Jam for Kids

The children raffled prizes from Universal Music Group. (with Ina Samir)

Game Jam for Kids

Car racing for fun. The cars are controlled with a Windows phone.

Game Jam for Kids

Game Jam for Kids

Of course what’s an event at the Microsoft Office without a game on the Xbox.

Game Jam for Kids

Thank you to all of you, sponsors, volunteers, participants and teachers!

Nia Angelina Samir

Yours Truly

Print Friendly

ASP.NET MVC Training

November 2014

I took 1 week training “Developing ASP.NET 4.5 MVC Web Applications – 20486” and I liked it. It’s an instructor-led class but it’s all done remotely. I was alone in an small office. Technically I don’t need to be in the course provider’s office but I find it convenient. It was my second class with ONLC and with this format of training. Everybody is remotely and all around US. Because of the East Coast participants the classes start way too early but they end early too. This time my class was Monday to Friday, 7AM-1:45PM PST. The class instructor MJ Parker was in NYC and she did a fabulous job.

MVC Training

There were 11 other participants. Some of them I wish I met in person and at least one I wish was never part of the class.

What I liked about the class:
– I like the convenience. This format offers a lot of freedom.
– You get the best instructors as they are not limited by the geographical location.
– I find it amazing that you can take the class as many times as you like in a year.

What can be improved about the class:
– There were way too many technical difficulties with the remote machines, conference bridge, Internet (the building I was in lost Internet for an hour or more and of course that means no phone, no labs, no instructor).
– The class format doesn’t facilitate you to network with the other participants and I’ll never know or connect with the people I spent a whole week on the phone with.
– Some participants are encouraged by the impersonal setting of the class to ask shameless and stupid questions.

This time I got a printed book from the class and I needed a lot of coffee. I love the book. I hope to get some time around Christmas to look at it some more.
MVC Training

The class doesn’t just cover designing and developing models, controllers, views for the web applications.

Here are the highlights on what else the class covered:
– Testing and Debugging.
– Front end development with CSS.
– Security.
– Using JavaScript and jQuery.
– Using SQL Azure.
– Creating/consuming Azure Web Services.
– Calling and developing Web APIs.
– Deploying web applications.

You get 1 month of free Azure account with the class. A lot of fun.
MVC Training

The bottom line is I loved it. Unfortunately because of the remote setting I was still able still to work half of the time and not to pay attention to the class but I’m thankful for the class and the knowledge.

Print Friendly