The word "core" in Joomla! pertains to the distributed files which are needed to create and administrate a Joomla CMS powered web site. These files can be downloaded from the Joomla website at http://www.joomla.org/download.html. The Joomla "core" also contains some basic functionality to get new Joomla installations working quickly and easily. Included are the user manager, article manager, weblink manager, category manager, contact manager, and menu manager. There is also a template manager with a few basic templates to power the front-end (website/user) view, a module manager with basic modules, plugin manager with basic plugins, and a few other "out of the box" extensions to extend the functionality of a basic Joomla installation. These core's extensions should not be confused with extensions which are available for downloading from the JED (Joomla! Extension Directory).
Reference: http://docs.joomla.org/Glossary